Evaluation of HPC Storage Systems for HEP Analysis

EVALUATION OF HPC STORAGE SYSTEMS FOR HEP ANALYSIS August 2024 AUTHOR(S): Asal Mehrabi CERN-ROOT SUPERVISOR(S): Jonas Hahnfeld Vincenzo Eduardo Padulano CERN openlab Report ABSTRACT While High-Performance Computing (HPC) systems are not yet critically relied upon for HighEnergy Physics (HEP) analyses, their role is expected to become increasingly important in the future. This project evaluates various HPC storage solutions, with a focus on the RNTuple data format, in terms of their performance, scalability, and suitability for processing the vast amounts of data generated by particle accelerators like the Large Hadron Collider (LHC). The analysis provides insights into how HPC systems might support and enhance future HEP workflows. EVALUATION OF HPC STORAGE SYSTEMS FOR HEP ANALYSIS 1 CERN openlab Report TABLE OF CONTENTS 1 Introduction 3 1.1 IntroductiontoRNTuple............................... 3 1.2 High-Performance Computing Systems . . . . . . . . . . . . . . . . . . . . . . . 3 1.2.1 LUMI ..................................... 3 1.2.2 Eiger...................................... 4 1.2.3 Job Scheduling with SLURM . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 Workflow ....................................... 4 1.3.1 DataPreparation............................... 4 1.3.2 Execution ................................... 5 1.3.3 PerformanceAnalysis............................. 5 2 Testing and Benchmarking 6 2.1 Multithreadedanalyses................................ 6 2.1.1 ImpactofStorage............................... 6 2.1.2 LustreAnalyses................................ 7 2.2 Analysis on Multiple Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.1 Multiple Concurrent Analyses . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.2 Comparison of Node Configurations and Concurrent Jobs . . . . . . . . . 8 2.3 Multithreaded vs Distributed RDF . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3.1 Implementation on HPC Systems . . . . . . . . . . . . . . . . . . . . . . 10 2.3.2 SLURMClusterSetup ............................ 10 2.3.3 Interactive Session on LUMI . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3.4 Results and Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3 Conclusions & Further Works 12 4 References 13 EVALUATION OF HPC STORAGE SYSTEMS FOR HEP ANALYSIS 2 CERN openlab Report 1 Introduction Although High-Performance Computing (HPC) storage systems are not yet critical for the processing of vast datasets produced by particle accelerators like the Large Hadron Collider (LHC), their importance is expected to increase significantly as data volumes grow. The ability to manage and analyze this data effectively is vital for advancing high-energy physics (HEP) research. In this project, some HPC storage solutions are systematically evaluated to assess their performance and scalability, with a specific focus on their suitability for HEP analyses. The primary goal of this project is to use the Analysis Grand Challenge as a benchmark for running high-energy physics (HEP) analyses on High-Performance Computing (HPC) systems, utilizing ROOT’s RDataFrame with data converted into the RNTuple format. This approach is essential, as RNTuple is expected to become the primary event storage format for the HighLuminosity LHC (HL-LHC), where it will be responsible for storing multiple exabytes of data. Successfully implementing RNTuple in this context is crucial to ensuring that HPC systems can meet the massive data processing demands of future HEP experiments, enabling them to support advanced research across the field. 1.1 Introduction to RNTuple RNTuple is the latest data storage format developed by CERN, designed to meet the complex demands of large-scale data management in high-energy physics (HEP). Utilizing a columnar format that efficiently supports complex and nested data structures, RNTuple significantly enhances data management by enabling faster processing, optimized memory usage, and improved overall performance. When compared to its predecessor, TTree, RNTuple introduces several key advancements that address the limitations of older storage models. While TTree is also a columnar format and has served high-energy physics (HEP) data storage needs for decades, its design does not fully leverage modern storage hardware and parallel computing capabilities. RNTuple, on the other hand, is designed with these considerations in mind, making it more suited for the increasingly complex and data-intensive workflows of future HEP experiments. Given these advancements, it is crucial to evaluate how RNTuple performs on HPC systems, which are central to modern data-intensive applications. Assessing RNTuple’s performance on HPC infrastructure will provide valuable insights into its efficiency and scalability in handling the massive data volumes expected from the High-Luminosity LHC (HL-LHC). This evaluation is essential for ensuring that RNTuple can meet the rigorous demands of future HEP experiments and effectively manage the anticipated exabyte-scale datasets. 1.2 High-Performance Computing Systems For this benchmarking, high-performance computing (HPC) systems were employed to evaluate the performance of data processing tasks. LUMI and Eiger were chosen as exemplary systems available during this project, with LUMI specifically selected for its flash-based storage. In this section, we examine the key characteristics of these HPC clusters and the SLURM job scheduling system used to manage and optimize computational resources. 1.2.1 LUMI LUMI, located at CSC in Finland, is an HPC system with over 200,000 cores spread across 2048 nodes, each with 128 cores. For this project, we utilized both the LUMI flash storage EVALUATION OF HPC STORAGE SYSTEMS FOR HEP ANALYSIS 3 CERN openlab Report and LUMI scratch storage. The flash storage, which uses SSDs, provides faster access times compared to the scratch storage that uses spinning disks. However, due to the higher cost of SSDs, the quota on flash storage is more limited. 1.2.2 Eiger Eiger, hosted at the CSCS in Switzerland, is another high-performance computing system, featuring around 50,000 cores, also distributed across nodes with 128 cores each. Eiger’s storage system, which is capable of managing multiple petabytes of data, integrates a parallel file system for efficient data access and high-throughput operations 1.2.3 Job Scheduling with SLURM Both LUMI and Eiger utilize SLURM for job scheduling and resource allocation. SLURM provides efficient job scheduling, managing the scheduling of jobs across the HPC clusters. It optimizes the order and allocation of computational tasks based on resource availability and job priority. Additionally, SLURM handles resource allocation by assigning computational resources such as CPUs, memory, and storage, ensuring the efficient use of the available infrastructure. 1.3 Workflow In this section, we will provide an overview of the workflow followed during the benchmarking process. As it is ilustrated in figure 1, the workflow encompasses several key stages, starting with the initial step of gaining access to the high-performance computing (HPC) systems, followed by data preparation, execution, and performance analysis. Figure 1: Workflow Steps 1.3.1 Data Preparation The first step in our workflow involves preparing the data and setting up the necessary software environment. Prior to commencing the benchmarking, it is essential to become familiar with the specific configurations, node types, and resource management policies of both LUMI and Eiger. Further on, we follow the following steps: EVALUATION OF HPC STORAGE SYSTEMS FOR HEP ANALYSIS 4 CERN openlab Report 1. Building ROOT from Source: ROOT is built from source on both HPC systems. This involved obtaining the source code from the official GitHub repository, configuring the build with CMake, and compiling with ‘make‘. The installation is completed in the designated directory. For detailed instructions, refer to the ROOT website: ROOT Installation Instructions. 2. Familiarizing with the Analysis Grand Challenge (AGC): Understanding AGC requirements and structure is crucial for aligning the benchmarking with project objectives. Detailed guidelines and datasets are available at AGC Documentation. 3. Using the NanoAOD Dataset: The CMS NanoAOD Open Data is used for the benchmark which was already converted into the RNTuple format. 1.3.2 Execution The execution phase involved submitting and managing computational jobs on the HPC systems. After preparing the environment and data, the benchmarking programs are submitted to the HPC systems (LUMI and Eiger) using SLURM. When submitting jobs, it is recommended to include the following in the bash script for launching the job: •To ensure reproducible results, it is recommended to launch jobs using the srun command. In this way, srun provides a well controlled execution environment, improving the reproducibility for multithreaded runs. •After some runs, we observed that job runs on LUMI and Eiger were faster on the second execution due to OS file caching and this was because cached files were accessed from memory, not from Lustre, causing inconsistent performance. The fadvise command addresses this by telling the OS to drop files from the cache, forcing the system to read them from Lustre. This ensures consistent performance by reflecting the true I/O times. 1.3.3 Performance Analysis Once the jobs are completed, the following steps are taken: •Collecting Output: The output data generated from the jobs is collected from the HPC systems. This data includes the execution time which is the data we use for further analyses. •Performance Analysis: A thorough performance analysis is conducted to assess various metrics such as processing speed, efficiency, and resource utilization to evaluate the effectiveness of the RNTuple format and the overall performance of the HPC systems. EVALUATION OF HPC STORAGE SYSTEMS FOR HEP ANALYSIS 5 CERN openlab Report 2 Testing and Benchmarking In this section, we conducted various analyses to evaluate the performance and characteristics of running RNTuple on high-performance computing (HPC) systems. The primary goal was to assess how well RNTuple handles large-scale data processing and to understand its efficiency and suitability for HPC environments. 2.1 Multithreaded analyses 2.1.1 Impact of Storage In this analysis, we evaluated the impact of different storage systems on execution time and speed-up. Specifically, we compared Flash Storage and Scratch Storage from LUMI, alongside the storage system from Eiger. As mentioned before, Flash storage utilizes SSDs which offers significantly faster access times. The results, illustrated in Figure 2, show that Flash Storage delivers the best performance in terms of execution time, followed by Scratch and then the Eiger storage system. Figure 2: Impact of Storage on Execution Time Following this, we analyzed the speedup behavior across these three storage systems. The speed-up was calculated relative to the performance at 16 cores. The result can be seen in figure 3and shows that all three systems scale well with the number of cores up to 64, with their speed-up closely following the ideal line. However, as we can observe, there is no significant improvement in terms of speedup after transitioning from half-node to full-node usage, indicating that the addition of more cores beyond a certain point does not yield proportional performance gains. For this reasons, in the following sections, we will delve deeper into this issue to better understand the factors contributing to it. EVALUATION OF HPC STORAGE SYSTEMS FOR HEP ANALYSIS 6 CERN openlab Report Figure 3: Impact of Storage on Speed-up 2.1.2 Lustre Analyses In this analysis, we focused on optimizing execution times by adjusting the stripe count and stripe size parameters of the Lustre file system on LUMI. Striping is a technique used in the Lustre file system to distribute file data across multiple Object Storage Targets (OSTs), which can enhance I/O performance. By default, the stripe count is set to 1, meaning that data is stored on one storage server per file, and the stripe size is set to 2 MB. Figure 4: Lustre Parameters Analysis •Stripe Count (2): Increasing the stripe count to 2 did not result in significant improvements in execution times compared to the default configuration. The impact on performance was minimal. EVALUATION OF HPC STORAGE SYSTEMS FOR HEP ANALYSIS 7 CERN openlab Report •Stripe Size (4 MB and 512 KB): Both of these stripe size configurations demonstrated suboptimal performance, particularly as the number of cores increased. Adjusting the stripe size to either 4 MB or 512 KB led to decreased efficiency and longer execution times. The default configuration consistently provided the best execution times across all scenarios. This was especially evident as the number of cores increased, where the default settings outperformed other configurations in terms of efficiency and speed. Therefore, these findings suggest that the default Lustre parameters are better suited for the given workload and hardware configuration than the adjusted settings.This observation is illustrated in Figure 4. 2.2 Analysis on Multiple Nodes 2.2.1 Multiple Concurrent Analyses The primary objective of this analysis is to run multiple copies of the same analysis concurrently to simulate a multi-user environment. In this analysis, we aimed to run concurrent jobs across different numbers of nodes. Figure 5demonstrates the average execution times for various configurations using 128 and 64 threads on the LUMI and Eiger HPC systems. Across both thread configurations, the execution times show minimal fluctuation, indicating that performance remains relatively stable regardless of the number of nodes or tasks deployed. Notably, there is no significant improvement in execution time when transitioning from half-node to full-node configurations. Figure 5: Multiple Concurrent Analysis for Full Node and Half Node 2.2.2 Comparison of Node Configurations and Concurrent Jobs In this section, we conducted an analysis to understand whether there is any cache effect related to loading the dataset while running concurrent jobs. The results are illustrated in Figure 6, where various configurations were tested to observe their impact on execution time. The key observations from this analysis are: •128 Threads, 1 Node: The baseline configuration with 128 threads on a single node showed stable execution times, serving as a reference for comparing other setups. EVALUATION OF HPC STORAGE SYSTEMS FOR HEP ANALYSIS 8