Plasma-PEPSC D1.10 Updated Assessment of Developments and Revision of Scientific Challenge, Controlling Plasma-Material Interfaces with BIT

HORIZON JU Research and Innovation Actions HORIZON-EUROHPC-JU-2021-COE-01-01 European High-Performance Computing Joint Undertaking Plasma Exascale-Performance Simulations CoE 101093261 D1.10 Updated assessment of developments and revision of scientific challenge - Controlling Plasma-Material Interfaces with BIT WP1: Plasma Simulations - Codes and Grand Challenges Date of preparation (latest version): 30/06/2025 Copyright©2023 – 2027 The Plasma-PEPSC Consortium D1.10: Controlling Plasma-Material Interfaces with BIT 2 DOCUMENT INFORMATION Deliverable Number D1.10 Deliverable Name Updated assessment of developments and revision of scientific challenge - Controlling Plasma-Material Interfaces with BIT Due Date 30/06/2025 Deliverable lead UL Authors Stefan Costea (UL), Jeremy Williams (KTH), David Tskhakaya (IPP CAS) and Leon Kos (UL) Responsible Author Stefan Costea (UL) E-mail: [email protected]lj.si Keywords Heat Load, Collisions, Cross-Sections, EEDF, MPI, OpenMP, OpenACC, Hybrid Programming, Heterogeneous Computing Systems, Parallel I/O, In-Situ Data Visualization and GPU Offloading WP/Task WP1/Task D1.10 Nature R Dissemination Level PU Final Version Date 30/06/2025 Reviewed by Tilman Dannert (MPG) & Julian Lenz (HZDR) D1.10: Controlling Plasma-Material Interfaces with BIT 3 DOCUMENT HISTORY Partner Date Comment Version UL 05/05/2025 Skeleton version 0.1 UL 02/06/2025 First draft 0.2 KTH 04/06/2025 Final draft updated for internal review 0.3 UL 18/06/2025 Revised draft after internal review 0.4 KTH 23/06/2023 Final cleanup for submission 1.0 D1.10: Controlling Plasma-Material Interfaces with BIT 4 Executive Summary In this deliverable, we provide a comprehensive update on the BIT1 code, highlighting recent developments in both computational physics and code optimization. These advancements and findings, including hybrid parallelism, GPU offloading, alternative data layouts, and improved memory management, further strengthen BIT1’s readiness for large-scale deployment on emerging exascale systems. D1.10: Controlling Plasma-Material Interfaces with BIT 5 Contents 1 Introduction 6 2 Revision of the Scientific Challenge 6 3 Summary of Developments 7 3.1 PhysicsDevelopments ............................ 7 3.2 Hybrid MPI and OpenMP/OpenACC BIT1 . . . . . . . . . . . . . . . . 8 3.2.1 Porting BIT1 to Multi-core CPUs . . . . . . . . . . . . . . . . . . 9 3.2.2 Offloading BIT1 to GPUs . . . . . . . . . . . . . . . . . . . . . . 12 3.2.3 Accelerating BIT1 Asynchronously with GPUs . . . . . . . . . . . 16 3.2.4 Heterogeneous BIT1 Future Directions . . . . . . . . . . . . . . . 19 3.3 CodeDevelopments.............................. 21 3.3.1 DataLayout.............................. 21 3.4 In-situ Data Analysis & Visualization Developments . . . . . . . . . . . . 22 3.4.1 Python................................. 22 3.4.2 ParaView ............................... 23 3.5 SoftwareRelease ............................... 23 4 Revision of Gap Analysis 23 5 Revision of Roadmap 25 6 Conclusion 26 D1.10: Controlling Plasma-Material Interfaces with BIT 6 1 Introduction BIT1 is a 1D3V (1D in coordinate space and 3D in velocity space) electrostatic ParticleIn-Cell (PIC) Monte Carlo (MC) code. It is optimized for the 1D modelling of the tokamak edge plasma, more specifically the so-called Scrape-off Layer (SOL). BIT1 simulates plasma, neutral and impurity particle transport inside the SOL including their nonlinear interaction, as well as plasma-surface interaction processes [1]. The simulation geometry corresponds to the magnetic flux tube limited by the socalled inner and outer divertor plates. The divertors, representing heat resistant plates, are designed to withstand strong heat and particles fluxes coming from the core plasma. Hot plasma (e, D+), impurity (e.g C+m, W+n) and heat source correspond to the particle and heat transport across the separatrix. After the injection, plasma and impurity particles propagate along the magnetic field towards the divertor. The particles absorbed at the divertor plates can cause injection of secondary particles (secondary electrons, D, C and W atoms). These atoms interact with the plasma in a nonlinear way. Atoms reaching the radial boundaries of the system (i.e. the separatrix and the outer wall) are removed from the system. Impurity ions, C+m, W+n, are also removed from the simulation with the probability corresponding to the anomalous cross-field diffusion coefficient and cross-field gradient length, e.g. ∼1 cm. The strength of the particle and heat sources and the temperature of the incoming particles are adjusted to match the experimentally observed plasma density and electron temperature in the upstream SOL [2]. 2 Revision of the Scientific Challenge Usually it is assumed that the parallel transport is the most simple and well studied type of plasma transport. In the scrape-off layer (SOL) this is not the case: low collisionality of plasma, different inelastic and short time scale processes, and geometrical effects can cause deviation of parallel transport from the classical one. As a result, the classical transport model can significantly overor underestimate particle and energy fluxes to the divertor targets. For present-day tokamaks these uncertainties are not essential, but for ITER, where transient heat loads on divertor targets represent one of the greatest threats to target lifetime, they can lead to serious unwanted consequences. Hence, development of realistic kinetic models of the parallel transport in the SOL is of top importance [3]. A number of innovative techniques have been implemented in BIT1, such as the optimized memory management [4] and the nonlinear multi-step multi-particle null-collision method [5, 6]. BIT1 enables to simulate and predict plasma and heat loads to the divertor plates during the steady state and transient events in today’s and future magnetic confinement fusion devices [1, 7, 8, 9]. One of the recent results concerns simulation of the SOL of the future generation tokamak ITER [10]. Different atomic physics models have been considered by including 102, 104and 106 atomic processes. Simulations indicate that the latter is essential for proper calculation of the electron energy distribution functions: the first two models predict hot electron tails of the energy distribution function, while the last shows no such tail. These hot electron fractions lead to the unacceptably large divertor electron heat loads ∼16 MW/m2, while D1.10: Controlling Plasma-Material Interfaces with BIT 7 the last model shows only 1 MW/m2[11]. Thus, it has been shown that, in the end, kinetic effects do not significantly modify particle and heat divertor loads in ITER [6]. Such simulations require 50 and more million core-hours and represent extremely challenging tasks with a duration of multiple months. Due to this long waiting period, obtained profiling results in D1.1 were investigated and utilized for Exascale preparation. In this case, the estimate of the Figure of Merit (FOM) for the exascale BIT1 has not changed. The Figure of Merit [12] was given as: FOM = H·t0 H0·One-week-in-hours ≈9.0×104cores, where H= 5 ×1017 and H0= 2.6×1016 particle time steps are the sizes of the exascale and the original simulation, respectively, and t0= 7 ×106core hours is the clock time for the original simulation. Additionally, in deliverable D1.1, Figure 1 was reported and showed where further BIT1 optimizations should be carried out to maximize the performance gain. For Exascale platforms, we will need to apply these required improvements to enable such ITER simulations and modelling within 1-2 weeks. Figure 1: Percentage breakdown of the BIT1 functions where most of the execution time is spent for the ionization and sheath case, respectively. 3 Summary of Developments 3.1 Physics Developments BIT1 has introduced a new model of the collision operator called dressed cross-section model (DCSM) that enables the implementation of a large number of atomic transitions (106and more) in its simulations [6]. This upgrade allows to accurately simulate collisions in the target case at little to no additional computational cost. D1.10: Controlling Plasma-Material Interfaces with BIT 8 The DCSM consists of two steps: calculation of averaged cross-sections for excitation collisions from the ground state and introduction of effective cross-sections based on Collisional-Radiative Model (CRM) rate coefficients. The effective collision cross-sections derived from the DCSM show expected asymptotic behaviour: (i) For low plasma density, they reduce to the conventional cross-sections for single-step transitions from the ground state; (ii) the Maxwell-averaged rate coefficients are equal to the corresponding rate coefficients obtained from the CRM. In order to test DCSM for realistic applications and study effects of multi-step atomic transitions in tokamak plasma edge, a set of BIT1 SOL modelling have been performed for JET and ITER relevant plasma conditions. The simulations did not show any significant influence of multi-step transition processes in a JET relevant relatively hot divertor plasma. Contrary to this, the electron heat loads to the ITER divertor plates were strongly affected by the multi-step ionization and recombination processes in the divertor plasma and have to be accounted for in future simulations [6]. BIT1 code has been involved in interpreting modeling for different fusion devices (tokamaks), such as W erosion rates at JET [13], updated measurements of divertor electron temperatures during the ELMs at JET and COMPASS [14], predictive/interpretive modelings of liquid metal experiments at ASDEX [15] and COMPASS-U [16], predictive modeling of plasma-surface interactions in DEMO [17]. BIT1 contributions to better understanding of plasma parallel transport in the plasma edge have been acknowledged in recent review paper by the ITPA DivSOL group (the top international team for plasma edge study in magnetic confinement fusion devices) [18]. 3.2 Hybrid MPI and OpenMP/OpenACC BIT1 The current implementation of BIT1 has traditionally relied exclusively on MPI and lacked support for hybrid shared-memory parallel computing and GPU offloading capabilities. To address these limitations, this deliverable presents a substantial extension of the BIT1 code through the development of hybrid “MPI+OpenMP” and “MPI+OpenACC” versions. These implementations are designed to improve node-level performance and enhance scalability by leveraging multi-core CPUs and GPUs. Motivated by the findings in [19, 20], a detailed investigation of the BIT1 code structure was conducted. Due to the complexity of particle arrangement in the arrj function, optimization efforts have remained focused on the mover function (particle pusher), which has been identified as one of the most computationally intensive components of the code in Figure 1. As presented in deliverables D1.1 [12] and D1.5 [21], this work resulted in the first GPU-enabled version of BIT1, leveraging directive-based programming models in OpenMP and OpenACC, with the mover function offloaded to NVIDIA GPUs. Building upon this development and a scientific article by Jeremy J. Williams et al. [22], the BIT1 team implemented the first asynchronous multi-GPU execution of BIT1. This implementation utilizes OpenMP Target Tasks with the “nowait” and “depend” clauses, along with OpenACC parallel regions employing the “async(n)” clause, where n specifies the queue number for concurrent executions, to achieve overlapping computation and communication across multiple GPUs, thereby significantly improving scalability. In this deliverable, our aim is to report on the performance improvements and scal- D1.10: Controlling Plasma-Material Interfaces with BIT 9 ability gains resulting from the hybrid and GPU-accelerated implementations of BIT1. Asynchronous GPU programming has shown to significantly reduce the execution time of the mover function, enabling efficient parallel execution across multiple nodes. The work completed in [22] outlined key configurations where OpenACC parallel regions employing the “async(n)” clause provided the best short-run performance, while OpenMP Target Tasks with the “nowait” and “depend” clauses offered the best scalability for large-scale runs. In this work, we use the following four distinct HPC systems: Dardel, an HPE Cray EX Supercomputer (CPU partition, 1,278 nodes); NJ, an HPC system; VEGA, a petascale EuroHPC supercomputer (CPU/GPU partition, 960/60 nodes); and MareNostrum 5 (MN5), a pre-exascale EuroHPC supercomputer (GPP/ACC partition, 6,480/1,120 nodes). Unless otherwise noted, all BIT1 simulation experiments were executed on a full node using all available threads, as in the scientific articles by Jeremy J. Williams et al. [20, 22]. A brief summary of these systems are provided in Table 1. HPC Systems Dardel NJ VEGA MareNostrum 5 (MN5) Nodes 1278 (CPU) 1 (CPU / GPU) 960 (CPU) / 60 (GPU) 6480 (GPP) / 1120 (ACC) CPU AMD EPYC Zen2 AMD EPYC 7302P AMD EPYC 7H12 Intel Sapphire Rapids 8480+ Frequency 2.25 GHz 3.0 GHz 2.6 GHz 2.0 GHz Cores 64 32 64 56 RAM 256 / 512 / 1024 GB 256 GB 256/1024 GB 128 / 256 / 1024 GB GPU – 2 ×A100 4 ×A100 4 ×H100 Interconnect HPE Slingshot – HDR100 (CX-6) NDR200 (CX-7) Compiler GCC 11.2.0 GCC 12.2.0 GCC 12.3.0 GCC 12.3.0 MPI Library Cray MPICH 8.1.17 OpenMPI 4.1.5 OpenMPI 4.1.4 OpenMPI 4.1.5 Table 1: Distinct HPC systems used for all BIT1 experiments and performance results. 3.2.1 Porting BIT1 to Multi-core CPUs An in-depth investigation of BIT1’s execution time was conducted in [20, 22] and presented in deliverables D1.1 [12] and D1.5 [21], focusing on both intra-node and inter-node performance to assess the benefits of hybrid approaches that combine MPI with OpenMP / OpenACC, as opposed to using MPI alone. If otherwise specified, the number of OpenMP threads or OpenACC cores used is the one required to fill up the node. Figure 2 presents BIT1 total execution and optimized mover function times with 2 and 16 ranks per node for 1000 timesteps on NJ. Both hybrid BIT1 “MPI+OpenMP” and “MPI+OpenACC” versions reduce total simulation and mover function times compared to MPI-only, showing improved performance through parallel threading on multicore CPUs. Extending to longer runs on VEGA with 16 and 64 ranks per node over 20,000 timesteps (Figure 3), the hybrid BIT1 “MPI+OpenMP” version yields significant speedups: at 16 ranks, total simulation time drops from 845.69s to 394.33s (53.4% faster) and mover time from 302.02s to 92.69s (69.3% faster). The “MPI+OpenACC” version further reduces times to 355.85s (57.0%) and 89.61s (70.3%), respectively. At 64 ranks, total time decreases to 164.13s and mover to 66.91s; adding OpenMP threads cuts these to 42.07s (74.4% reduction) and a 37.2% reduction for mover. The hybrid BIT1 “MPI+OpenACC” with 64 ranks achieves the best performance, reducing total simulation to 32.32s (80.3% reduction) and mover to 32.32s (51.7% reduction) versus the 64-rank baseline. D1.10: Controlling Plasma-Material Interfaces with BIT 16 Figure 12: BIT1 Optimized mover function (strong scaling) execution time(s) on Vega for 200 time steps up to 16 Nodes (64 MPI Ranks and 64 GPUs) using OpenMP and OpenACC Unified Memory on NVIDIA GPUs. [22] 3.2.3 Accelerating BIT1 Asynchronously with GPUs Extending upon our earlier 64 GPU porting efforts, we enhance BIT1 with the first asynchronous multi-GPU implementation of the particle mover function using up to 128 GPUs. Leveraging OpenMP Target Tasks with “nowait” and “depend” clauses, along with OpenACC using async(n), and UM, we aim to improve execution through concurrency and simplified memory management. This asynchronous approach improves load balancing and resource utilization, positioning BIT1 to exploit exascale platforms effectively. Figure 13: NVIDIA Nsight Systems OpenACC UM / ACC Parallel View - GPU porting of BIT1 mover function on Vega with 1 time step. [22] D1.10: Controlling Plasma-Material Interfaces with BIT 17 Figure 14: NVIDIA Nsight Systems OpenACC UM / ACC Parallel Async(n) View - GPU porting of BIT1 mover function on MN5 with 1 time step. [22] Accelerating BIT1 Asynchronously with 128 GPUs. We evaluate this asynchronous strategy on the MN5 pre-exascale EuroHPC system, scaling to 32 nodes (128 GPUs) with 200 timesteps. As in previous tests, each configuration used 4 MPI ranks, with one dedicated GPU per rank to prevent resource contention. Profiling with NVIDIA Nsight Systems showed a significant reduction in the time consumed by the primary kernel on the GPU. This decreased from approximately 99.7% on NJ and Vega systems (Figure 8 and 13) to 58.6% on MN5 (Figure 14), due to the overlap of computation and communication. Figure 15: BIT1 Total Simulation (strong scaling) execution time(s) for 200 time steps up to 32 Nodes (128 MPI Ranks and 128 GPUs) using OpenMP and OpenACC Unified Memory on MN5 H100 NVIDIA GPUs. [22] For the total simulation (Figure 15), the OpenACC Async(n) version improved performance by 5.31% over OpenMP Nowait version on 1 node and showed minor gains up D1.10: Controlling Plasma-Material Interfaces with BIT 18 Figure 16: BIT1 Optimized mover function (strong scaling) execution time(s) for 200 time steps up to 32 Nodes (128 MPI Ranks and 128 GPUs) using OpenMP and OpenACC Unified Memory on MN5 H100 NVIDIA GPUs. [22] to 8 nodes. However, scalability declined beyond that, resulting in a 6.39% slowdown at 32 nodes. The CPU version remained faster across all node counts, although it experienced slight saturation beyond 16 nodes. For the mover function (Figure 16), the OpenACC Async(n) version outperformed OpenMP Nowait up to 4 nodes, achieving a 14.01% speedup on 1 node. Beyond 8 nodes, performance gains diminished, and OpenMP Nowait version became more effective due to better data movement efficiency. The CPU version continued to outperform both GPU versions, especially for larger runs, highlighting the need for further tuning to address scalability limits and communication overhead on large-scale systems. Extreme Scaling BIT1 Asynchronously with 400 GPUs. At extreme scale on MN5 with up to 400 GPUs, BIT1 shows both gains and limitations. The OpenMP Nowait version scales more efficiently than the OpenACC Async(n) version, particularly at high GPU counts (Figure 15 and 16). For the total simulation, OpenACC reduces runtime from 23.24s (32 GPUs) to 10.70s (400 GPUs), while OpenMP improves slightly more, from 23.11s to 10.43s. The mover function follows a similar trend, with OpenACC dropping from 6.17s to 0.97s and OpenMP from 6.11s to 0.80s. However, scaling beyond this point is limited by inter-node communication and synchronization overheads. On the CPU (GPP) partition, performance improves initially but degrades at higher node counts, with total runtime dropping from 3.96s (8 nodes) to 2.42s (16 nodes), then increasing to 8.74s (100 nodes), revealing scaling inefficiencies. Although the CPU is slightly faster at small scale, GPU versions offer better scalability potential with further optimization. D1.10: Controlling Plasma-Material Interfaces with BIT 19 Figure 17: BIT1 Total Simulation (strong scaling) execution time(s) for 200 time steps up to 100 Nodes (400 MPI Ranks and 400 GPUs) using OpenMP and OpenACC Unified Memory on MN5 H100 NVIDIA GPUs. [22] Figure 18: BIT1 Optimized mover function (strong scaling) execution time(s) for 200 time steps up to 100 Nodes (400 MPI Ranks and 400 GPUs) using OpenMP and OpenACC Unified Memory on MN5 H100 NVIDIA GPUs. [22] 3.2.4 Heterogeneous BIT1 Future Directions While GPU-accelerated versions of BIT1, particularly the 4 MPI + 4 GPUs OpenACC Async(n) configuration, show notable improvements, the CPU version remains fastest at smaller scales. As node counts grow, OpenMP versions like the 4 MPI + 4 GPUs OpenMP Target Tasks Nowait configuration scale more efficiently, making them more D1.10: Controlling Plasma-Material Interfaces with BIT 20 suitable for large-scale PIC simulations. However, data transfer bottlenecks between CPU and GPU during each PIC cycle, shown in [19, 20, 22], limit overall BIT1 simulation performance. Profiling results presented (Figure 8, 13, and 14) confirm these overheads. To mitigate these limitations, minimizing CPU to GPU data movement is essential, with reorganization of the original data layout [23] being a critical step toward improving performance and ensuring better scalability. Optimizing data structures and memory layout presents promise to accelerate BIT1 on GPUs. As shown in Figure 19, the function moveb() widely used in production cases (such as the High-Density Sheath Simulation [19]), currently stores particle properties in arrays, which increases memory usage and causes inefficient GPU memory access [23]. Switching to a vector of particle objects, moveb VoS(), allows dynamic resizing and easier attribute access on GPUs [24, 25, 26], enhancing scalability and adaptability. Using an array of particle structures, moveb AoS(), ensures contiguous memory access, improving throughput and reducing latency, which aligns better with GPU processing [27, 28]. Figure 19: An initial investigation of the data structure and memory layout for two of the most computationally intensive BIT1 functions on Dardel. [22] Integrating the Low-Level Abstraction for Memory Access (LLAMA) framework, a C++ library for multidimensional arrays and custom memory layouts at compile time [29] developed by one of the project partners, can further optimize BIT1’s memory access and accelerate GPU performance to better adapt to diverse computing environments and hardware. D1.10: Controlling Plasma-Material Interfaces with BIT 21 3.3 Code Developments 3.3.1 Data Layout Different data layouts have been tested on the most time-consuming functions in BIT1, namely moveb() and arrj()[19], for runtime speedup potential. The data layouts tested were manual implementations of flat 1-dimensional array, Iliffe vector and array-ofstruct (AoS) and various automatic implementations of the LLAMA library (Low-Level Abstraction of Memory Access). They were compared to the original BIT1 data layout, i.e. nested pointers, for various problem sizes. Differences in the manual implementations with static memory allocation: •nested pointers (original BIT1) – multi-dimensional array of pointers which point to the start of each small allocated contiguous memory of size max particles per cell. •flat – 1D array where the only pointer used points to the start of the large allocated contiguous memory of size number of species ×number of cells ×max particles per cell. •Iliffe vector – multi-dimensional array of pointers which point to addresses inside the large allocated contiguous memory of size number of species ×number of cells ×max particles per cell. We have found that most of the alternative data layouts tested here speed up the execution time of both moveb() and arrj() functions, for various problem sizes, except for the Iliffe vector and partially for the flat 1D array. The findings are shown in Figure 20. For the flat 1D data layout, the execution time of moveb() has increased approximately 4 times after a certain problem size (13k particles, Figure 20c) which is just above the size of the L2 cache of the CPU used for testing. A more detailed analysis has been done on the L2 cache access efficiency using the PERF Monitoring Tool: perf stat -M all_l2_cache_accesses,all_l2_cache_hits,all_l2_cache_misses \ ./exe Comparing cases (b) and (c) in Figure 20, where the problem size changes from just below to just above the L2 cache size, results show that case (c) has 2.5×more prefetch misses, nearly 30% less efficient prefetching (lower L2-only hits) and 20% more instruction/data cache (IC/DC) misses. Since particle data in BIT1 is statically allocated and grouped by cells, the particle arrays are strided with particle vacancies of length comparable to the average number of simulated particles per cell. If a large number of particles per cells is used, then a smaller number of cells per CPU must be used in order to ensure that the working sets fit into the L2 cache. The same slowness of moveb() using the flat 1D array data layout applies to the Iliffe vector data layout, where the array of pointers point to addresses inside the large contiguous allocated memory as in the case of the flat 1D array. Due to the pointer referencing of the multi-dimensional array of pointers, the Iliffe vector additionally inherits the slowness of arrj() from the original BIT1 data layout, making it the worst option for data layout in memory for BIT1. In a multi-core approach, i.e. CPU multithreading and/or GPU, the moveb() function is highly parallelizable, making the flat 1D array also a viable candidate for efficient D1.10: Controlling Plasma-Material Interfaces with BIT 22 (a) (b) (c) (d) Figure 20: Execution times for arrj() and moveb() functions using various data layouts, for different problem sizes: a) 3K, b) 12K, c) 13K and d) 600K particles per CPU. Note: different scales for left and right axes. The comparison is made between different implementations for the same function, not between functions with the same implementation. data layout in BIT1. Of all the tested data layout implementations, the flat 1D array and the Iliffe vector are the only ones that can be tested in-place without duplicating memory usage and copy-to-and-from operations for selected functions. In the long run, the LLAMA implementation seems to be the best since it is one of the fastest in our tests and it can change its data layout on demand, without the need for rewriting the whole code, making it highly adaptable to hybrid hardware. 3.4 In-situ Data Analysis & Visualization Developments 3.4.1 Python Progress has been made on the user-experience side using parallel I/O output. The visualization Python scripts have grown in code size and have been split into two categories: profiles and stats. The profiles script shows plasma profiles, i.e. physical quantities at cell center or cell D1.10: Controlling Plasma-Material Interfaces with BIT 23 edge, along a spatial coordinate. In the current version of the script, the following plasma profiles are plotted from the diagnostics output of BIT1: space potential (V), particle density (m−3) and temperature (eV) for each species. An example is shown in Figure 21a. The stats script gives information on: •statistics –profile of particles per cell –evolution of total particles in the simulation •particle load balance per MPI rank •numerical stability –plasma frequency test: ωp·∆t <0.2 –cell size test: ∆x < λDebye –fast particle test: 5 ·vth,e·∆t <∆x as shown in Figure 21b. The upper limits of the numerical stability tests are shown with red patches. 3.4.2 ParaView Parallel I/O output from BIT1 (.bp4) can also be viewed using ParaView version >5.12. An example is shown in Figure 22 where the Plot Over Line filter has been used on the multi-dimensional data. 3.5 Software Release The I/O branch has been upgraded to BIT1 version C9 and merged with the GPUOpenMP branch on the BIT1 Gitlab repository. The purpose of the merge is to maintain a single codebase for development. More development branches, especially from other GPU implementations, are soon to be merged and preprocessor directives are going to be used to switch between the different GPU implementations at compile time. The parallel I/O functionality in BIT1 can be enabled on demand at compile time by passing USE_OPENPMD=1 to the make command. This environment variable is then read by the source code and preprocessor directives decide the inclusion of the parallel I/O library and code at compile time. 4 Revision of Gap Analysis In the first deliverable the principal gap was the sheer computational cost of a full–fidelity kinetic simulation of the ITER scrape-off layer; the code required well over fifty million core-hours and would have occupied the entirety of even a large leadership machine for several months. Since then we have narrowed, but not yet closed, the distance between D1.10: Controlling Plasma-Material Interfaces with BIT 24 (a) (b) Figure 21: In-situ visualization of parallel I/O output: a) plasma profiles, b) statistics, load balance and numerical stability (upper limits shown with red patches). D1.10: Controlling Plasma-Material Interfaces with BIT 25 Figure 22: ParaView visualization of plasma profiles from parallel I/O output: top – electron density (m−3), bottom – electron temperature (eV). that requirement and practical turnaround times. The hybrid MPI + OpenMP/OpenACC versions now scale to hundreds of GPUs and many thousands of CPU cores, yet the end-to-end Figure of Merit still exceeds the threshold that would allow an ITER discharge to be completed within one to two weeks of wall-clock time. The dominant sources of inefficiency are known: frequent CPU-to-GPU data transfers caused by the legacy particle arrays, the absence of overlap between communication and computation in the main loop, and the fact that only the particle mover has so far been ported in depth to accelerators while arrj() and several collision kernels remain CPU-bound. The gap has narrowed and closing the remaining distance requires integrating the proven cache-aware data layouts throughout the code and finishing the accelerator port of the arrj() function and of the collision operators. Addressing these points will bring BIT1 to the level at which routine, week-long kinetic runs of ITER-scale plasmas become a reality. 5 Revision of Roadmap Since the previous deliverable we have moved much closer to exascale readiness, but several focused tasks still lie ahead. Optimizing GPU memory management remains a priority, as further reductions in CPU to GPU data transfers and associated latency are critical to eliminating the remaining performance bottleneck in the accelerated code path. In parallel, the alternative data layouts that already demonstrated superior cache behaviour during benchmarking must now be subjected to large-scale performance and correctness tests, with particular attention to the most demanding kernels such as moveb() and arrj(). The current multi-GPU prototype also has to be deepened into a fully fledged implementation in which computation and communication are overlapped; doing so will provide the parallel efficiency required when BIT1 is deployed across hundreds of devices. Finally, systematic experimentation with OpenMP target off-loading constructs will allow us to benchmark this approach against the existing OpenACC pathway and