Collaborative development of a performance analysis methodology

Collaborative design of a performance analysis methodology Thomas Flynn (RSE) Advanced Research Computing Durham University Outline We will touch on: The motivation for this work Overview of our performance assessment The current state of our performance assessment Outlook and todo list Performance analysis Here we refer to performance analysis as the study of determining performance issues or bottlenecks in software that is typically running on High-Performance Computing (HPC) systems We are not concerned here with performance engineering - software development to amend performance issues Durham Performance Analysis Workshops 2025 Motivation 1: we all want faster code We use compute resources more efficiently - moving jobs through queues faster! It should allow researchers to do more research More energy efficient: more science per unit of emissions Motivation 2: accelerators Increasing availability of GPUs Potential novel hardware, particularly informed by ML/AI workloads Do we know which hardware suits our code? Motivation 3: performance analysis as a part of RSEing Many RSEs and other Research Technical Professionals (RTPs) engage with performance analysis - knowledge is very spread out Performance assessments can create more RSE opportunities Could we establish best practices for the community? The problem We have lots of performance tools, but they can often be overwhelming for new users - generate lots of data! There are standard performance issues, e.g., load imbalancing and poor use of hardware but do we know how to isolate these issues Conversations with researchers and developers show that tool-focused workshops can often lead to little implementation of these methods Our proposed contribution Instead of being driven by performance analysis tools, we propose developing a methodology which guides our choice of performance analysis tool Performance report Inter nodeIntra node Core GPU I/ O 80400100 60 20 High-level analysis: overview We split performance into 5 performance topics 1. Core 2. Intra-node - typically via OpenMP 3. Inter-node - typically via MPI 4. GPU 5. I/O Most of these topics cannot be decoupled High-level analysis: core We split core analysis into memory bandwidth and compute rate (FLOPS) which we can measure via LIKWID likwid-bench -t triad -W S0:2GB:1 likwid-bench -t peakflops -w S0:16kB:1 These microbenchmarks give the theoretical peak, then we want to see how our code ./my-exe compares likwid-perfctr -f -C 0 -g MEM ./my-exe likwid-perfctr -f -C 0 -g FLOPS_DP ./my-exe High-level analysis: core metrics For core performance we have memory bandwidth ratio and peak FLOPS ratio High-level analysis: Jupyter notebook A high-level performance report for the Cloverleaf benchmark is implemented through a Jupyter notebook High-level analysis: intra-node For intra-node analysis we use a strong scaling analysis, i.e., increase core count for a fixed problem size for i in 1 2 3 4 do env OMP_NUM_THREADS=$i time -f "%e" -o time-cores-$i.log ./my-exe done where total runtime is our observable High-level analysis: intra-node (alternative) The previous method can suffer from low thread affinity so we use likwid-pin for i in 1 2 3 do likwid-pin -c N:0-$i ./my-exe done Separate this into a SLURM script and save timing data to output file for a simple strong scaling analysis High-level analysis: intra-node metrics We use parallel efficiency and then our intra-node metrics are based on core counts we therefore evaluate performance with High-level analysis: Jupyter interface We compare to serial (with no OpenMP!) Yellow dashed line - below this core count parallel efficiency drops below 80% Red dashed line - below this core count parallel efficiency drops below 60% High-level analysis: Jupyter interface Rapid initial decrease in runtime Slower time decrease for high core counts Peaks at and High-level analysis: Jupyter interface Our intra-node metrics show very poor scaling Todo list Finalise both the high and low-level analyses for inter-node, I/O and GPU Lay out our performance analysis methodology, start-to-finish, in a whitepaper Put full methodology into practice for multiple 'real-world' codes Use in conjunction with hardware testbeds SHAREing This work falls under the SHAREing grant See the SHAREing webpages here: https://shareing-dri.github.io/ One aim of this grant is to incorporate this evolving methodology into a Performance Assessment service Thank you! This work is in collaboration with Tobias Weinzierl and has benefited from discussions with colleagues in ARC and the Scientific Computing group of Durham University Special thanks to our work experience student Ben who did most of the work on the Jupyter notebook This project has received funding through the UKRI Digital Research Infrastructure Programme under grant UKRI1801 (SHAREing) Results presented here were generated via Durham University's Hamilton HPC service