3-D seismic wave propagation modelling from the fault to the site for realistic earthquake scenarios SEM3DonGPU

Demande d’Attribution de Ressources Informatiques Description scientifique du projet 1 Informations g´en´erales Titre du Projet : 3-D seismic wave propagation modelling from the fault to the site for realistic earthquake scenarios SEM3DonGPU. Num´ero du dossier : AD010414346 (TMP29489) Responsable scientifique : GATTI Filippo Structure de recherche : Universit´ e Paris Saclay, CentraleSup´ elec, CNRS, ENS Paris-Saclay, Laboratoire de M´ ecanique Paris-Saclay UMR 9026 Nombre d’heures demand´ees sur le projet : IDRIS HPE Jean Zay A100 : 30000 heures GPU 2 R´esum´e This project aims at testing the high-fidelity 3-D earthquake simulator SEM3D (patented [1, 55] and open source) on GPU. SEM3D simulates an earthquake in solid-fluid domain, from active faults to free surface, across urban regions ≈100 km ×100 km ×100 km large, as sketched in Figure 1. The objective of this proposal is to extensively test the newly developed GPU version of the earYPO(f) = E(M;f)·P(R;f)YPT(f) = E(M;f)·P(R;f)·G(f) FIGURE 1 – Sketch of the source-to-structure earthquake simulations at regional scale. E(M;f): earthquake source (active fault), depending on the magnitude Mand the frequency content f(limited by the size of the mesh); P(R;f): scattering/attenuation effect on the earthquake wave field due to propagation path Pwithin the Earth’s crust ; G(f): non-linear site-effects induced by the shallow softer soil layers on the propagating elastic waves. PT: monitoring point at free surface, close to structural components (near-field) and polluted by the wave field scattered by the soil-structure interaction; PO: monitoring point at free surface, on hard bedrock, sufficiently far away from the structural components, supposedly neither influenced by site effects nor by structural back-scattering. thquake engine, namely SEM3D-GPU, to assess : 1 1. the feasibility of adopting SEM3D-GPU to participate to the international SMATCH benchmark 1, whose goal is to perform blind stress tests of isolated Nuclear Power Plant (NPP) submitted to realistic earthquakes; 2. the benefit of using SEM3D-GPU to extend the open-source synthetic earthquake datasets HEMEW-3D and HEMEWS-3D 2that we produced in 2023, in collaboration with the CEA (and leveraging TGCC resources). The scientific interests behind the latter two points are multifold. Extending the existing open access databases of earthquake ground motions (with corresponding geology and source) will be vital to train advanced machine learning and artificial intelligence surrogate models, such as our recent F-FNO (Factorized Fourier Neural Operator) described in [2]. The goal is to assess the computational cost of extending HEMEW-3D and HEMEWS-3D, which include more than 30000 instances each 3, but that would benefit from the addition of new realistic scenarios, accurate on a larger frequency range and covering larger regions, possibly including topography instead of flat surface. On the other hand, we need to assess the I/O best practices to unleash the increased speed-up and size-up of SEM3D-GPU, whose performance bottleneck seems to be related to the I/O operations to store the results. This aspect is crucial from a scientific standpoint and it demands some further testing and development. The allocation will steer the choice of the a trade-off between speed-up and snapshotting frequency when dealing with large test cases. 3 Pr´esentation g´en´erale 3.1 General Context Earthquakes are one of the most complex natural phenomena for scientists to understand and predict. The cogency of improved seismic hazard assessment (SHA) is motivated by the high uncertainty on the originating mechanisms, coupled with their disaster potential, increased portfolio exposure and structural demand. SHA can be assessed by either following a deterministic approach (DSHA), or by either considering a fully probabilistic approach (PSHA). While the former consists into a conservative direct modelling of the most intens ground shaking scenario,the latter bears on the statistics drawn from previous strong ground motion observations) [3]. In this context, many historical strong ground motion events have been successfully reproduced by employing physics-based numerical simulations (PBSs) [4, 5, 6, 7, 8, 9, 10, 11, 12]. PBSs provide great insights on the intricacies of earthquakes’ anatomy, characterizing the 3-D seismic wave motion in near-fault conditions (including possible directivity effects) and including the complex interactions with the surface topography and the geological settings (surface waves, wave trapping within basin-like structures). This achievement has been made possible by the development of high-fidelity/high-performance numerical codes, successfully deployed on large supercomputer infrastructures. As an example, an impressive result was obtained by a Chinese research group, who run an non-linear earthquake simulation over a continental 320 km by 312 km by 40 km region, up to 18 Hz [13]. The use of octrees has made possible a high degree of scalability in mesh generation using up to 220.000 cores [14]. Recent studies [15, 16] have paved the way to the use of PBS, traditionally employed for DSHA, in PSHA too, via optimized scheduling and scenario sampling techniques. Moreover, high-fidelity PBS widened the research horizons concerning the 1. Seismic Base Isolated Nuclear Power Plant Shaken by a Real Earthquake (SMATCH), see https:// smatch-benchmark.org/. The benchmark is organized by IRSN (Institut de Radioprotection et de Sˆ uret´ e Nucl´ eaire) and EDF (´ Electricit´ e de France), under the umbrella of OECD-NEA (Nuclear Energy Agency). 2. Available at recherche.data.gouv.fr. 3. In HEMEW-3D, only the variability on the geological setups are considered, whereas in HEMEWS-3D, both variability on geology and source location/characteristics are investigated. 2 study of complex features such as the attenuation and coherency of coda wave in the seismic signals [17], the directivity and the incoherence of the near-source wave motion [18] and non-linear site effects [19] among others. The advent of exascale computing is fostering the development of integrated end-to-end earthquake simulators and seismic digital shadows 4for critical structures and infrastructures [21], capable of provide reliable and quasi-real-time seismic risk prediction and structural response, as well as updated seismic hazard maps for regulatory public policies. In areas of moderate seismicity, such as metropolitan France, a general lack of direct seismic observations limits clashes with the demand of updated seismic safety margins for critical structures and infrastructures (nuclear power plants and dams especially). Given the fact that the majority of them has been constructed a few decades ago, French authorities face the urgency of a general reassessment of the seismic safety margins foreseen at the design stage, exploiting the gathered knowledge in seismology, earthquake engineering, the new geophysical and geological data, as well as the new technological tools available. Nuclear facilities, dams and other important infrastructures must undergo extensive stress testing, to assess and update their status and functionality, in case an unexpected (at the design stage) earthquake occurs. Moreover, those stress tests must include complex soil-structure interaction that were neglected in the past (surface waves, near-field conditions) in favor of more conservative design choices. This commitment fosters the definition of site-specific studies (the Cadarache nuclear site for instance) which can now be virtually tested via high-fidelity digital shadows. This is the general context of the SMATCH international benchmark focusing on a seismic base-isolated Nuclear Power Plant (NPP) submitted to a realistic earthquake. One major obstacle hindering the extensive use of synthetic ground shaking response in structural design is PBS limitation to a relatively low frequency interval, reaching maximum accuracies beyond 1 Hz, but up to 10 Hz. However, the vibration footprint of critical above-ground structures and related non-structural system components (such as piping systems and other equipment) are characterized by resonant eigenmodes way above 5 Hz [22]) and up until 40 Hz. The reasons behind this discrepancy between fault-to-site and site-to-structure simulations are two, namely : (i) the increasing computational demand implied by finer non-aliasing spatial discretization, required to simulate wave interactions at shorter wavelengths; (ii) the large uncertainty on the underground geological and seismotectonic configuration (the mechanical properties of huge chunks of Earth’s crust, including shallow soil non-linearity [23], the geometry of the geological interfaces between layers [24], the geometry of active fault segments, their roughness, properties and tectonic stress acting on them before the earthquake occurs), preventing the accurate calibration of the numerical model. The project proposal is meant to be a follow-up of 2023 and 2024 IDRIS Open Hackathons on GPU, that allowed to port the earthquake engine, SEM3D on GPU. SEM3D-CPU has extensively been used in previous DARI allocations, allowing to explore regional earthquake scenarios for different sites of interest, among which the experimental site of Argostoli [27, 28] and the Cadarache ITER nuclear facilities, located in the tectonic region dominated by the system of active faults of Middle Durance (MDF), in South-Eastern France [24, 29]. The latter model was constructed and validated in allocation A8 : two models were constructed, with different mesh refinement (see Table 1). However, those studies were performed on CPU nodes, with the CPU version of SEM3D-CPU, while waiting to stabilize its GPU version, started during the 2023 edition of the Open Hackathon. Dynamic access TMP29489 (issued after the 2023 Hackathon) was meant to complete the SEM3D-CPU porting on GPU, but it had partially been exploited differently, due to code-managing issues. Thanks to Open Hackathon 2024 edition, SEM3D-GPU can now be used in more realistic test cases, 4. According to [20], in a digital shadow, data flow is unidirectional from physical to digital twin. 3 tag No. □Elements [106]GLL DOFs [109]fmax #1 ≈3.8 5 ×5×5≈1.44 5 #2 ≈13.2 5 ×5×5≈4.94 10 TABLE 1 – Characteristics of the Cadarache numerical models constructed and validated with the computational resources of this allocation. which, according to us, justifies the extension of the dynamic allocation we ask for in this proposal. Albeit the outstanding results obtained by high-performance computing in physics-based modeling, due to its ontological determinism and to the large computational costs at stake, the methodology is still hardly adaptable to tackle large parameter sweeps, provided with the mentioned high level of uncertainty, convolved with the numerical dispersion. Our first attempt in this sense was made with HEMEW-3D and HEMEWS-3D datasets [25], where we simulated 0-5 Hz accurate earthquake scenarios over a rather small 9 km ×9 km ×9 km region. This endeavour allowed explore how the source and geological uncertainty propagates through the model and how to attempt at reducing it (the choice of eigen-geologies [26]) and to quantify seismic risk assessment of critical structures by exploiting synthetics time histories. 3.2 Perspectives on AI-assisted physics-based earthquake generation on JeanZay The activity report shows how we could successfully make use of part of the dynamic allocation, to develop complementary AI tools that SEM3D-CPU simulations could feed (see Rapport d’activit´e). Among those AI tools, the dynamic allocation focused on developing a Generative Adversarial Network framework for complementing the 5-30 Hz frequency spectrum of a SEM3D-CPU simulation (Gottfried Jacquet’s PhD Thesis). The dynamic allocation allowed to test several GAN architectures (for a total approximately 440M parameters for the generators and 220M parameters per discriminator), such as ALICE (Adversarially Learning Inference and Cross Entropy) [30], Pix2Pix (signal-to-signal translation), BicycleGAN (Bidirectional Cycle Generative Adversarial Network) [31], and MUST-ALICE (Multi-modal Unsupervised Signal Translation ALICE, shown in Figure 2), in order to be able to achieve super-resolution earthquake generation in a one-tomany framework (a SEM3D-CPU simulation, represented by 3-components time histories x(t)(valid in 0-1 Hz frequency range) for several seismograms at broadband 0-30 Hz y(t). The results presented in Figure 2 were obtained thanks to an improved training stability of the GAN framework, on ≈128000 seismic recordings of 4096 time steps each. This achievement was granted by the possibility of exploiting a larger number of GPU per job (under a Distributed Data Parallel paradigm) on Jean-Zay and a mini-batch size 4 times larger than the feasible mini-batch size on Universit´ e Paris-Saclay’s cluster (16 signals maximum). Jean-Zay’s GPU resources allowed us to specifically address the open question of disentangling the latent variables representing an encoded version of the earthquake ground motion time series, either simulated via PBS, either recorded. 3.3 Research perspectives for this proposal According to what stated in the summary (see Section 2), the present proposal is devoted to the research goals detailed in the following paragraphs. 4 (a) (b) (c) (d) (e) FIGURE 2 – Example of result for the Multi-modal Signal Translation. (a) List of time-frequency Envelope and Phase Goodness of Fit (EG and PG respectively, see [32]) for the MUST architecture. The target is represented by the recorded earthquake time histories at each station, namely y(t)(black signals). The MUST-ALICE input is the PBS simulation (obtained, for instance, via SEM3D-CPU) (b) One-to-many super-resolved earthquake time histories, obtained from SEM3D-CPU simulation xand with MUST-ALICE generative framework. xis encoded by a deep conformer architecture Fxy, branched into a disentangled latent representation : Fc xy for the common features to both PBS x(t)and records y(t), the other for the high-frequency features of y. The latent representation is decoded by Gyresidual decoder with conformer and AdaIn layers. (c-e) Variants of the generated signal Gy(Fc xy(x),N(0, I)), where the high-frequency features of yare sampled from white noise. SMATCH benchmark So far, very rare are the examples of complete fault-to-structure interaction studies, with strong Soil-Structure Interaction coupling (see for instance [33, 34, 35, 21]). In allocation A8 and A10, PBS has been used for models #1 and #2 in Table 1, improving to be able to properly model the interactions between the basin-like sedimentary deposits underneath the ITER facility at Cadarache and the input wave motion, generated by the nearby extended fault (≈14 km ×7 km) [36]. In last years, we developed (in collaboration with EDF R&D) a pipeline chaining scheme to use SEM3D-CPU regional simulations as input motion for commercial code of 5 structural dynamics (i.e., code aster, developed by EDF itself) and to simulate a fault-to-structure earthquake scenario with the so called Domain Reduction Method (DRM, see Figure 3), proposed by Bielak et al. [37]. It consists into a two-step analysis, declined as follows (see Figure 3) : (i) regional scale wave-propagation in simplified geological medium (typically the deep viscoelastic stratified Earth’s crust), not including the non-linear site-effects; (ii) a second run of the analysis on a smaller domain (eventually adding the structure) delimited by an artificial boundary at which equivalent inertial forces and free-field velocities, obtained in the precedent step, are applied. [38] applied this method to study the structural response of the Kashiwazaki-Kariwa Fault Ωs Ω′ s Ωsnl Γ (a) Fault Ω0 s Ω′ s Γ (b) Ωs ˆ Ω′ s Ωsnl Γ ˆ Γ (c) FIGURE 3 – Schematic representation of the domain reduction method (adopted by [37]). (a) Reference problem ; (b) backbone regional problem; (c) reduced domain problem. Nuclear Power Plant (KKNPP) in a fault-to-structure framework, up to 1 Hz. [39, 21] developed the EQSIM platform for exascale computing of structural response via DRM. Thanks to the collaboration with EDF R&D and previous allocations A8 and A10, we managed to reproduce realistic fault-to-structure earthquake simulations with both code aster and Cast3m. One major goal of this proposal is to employ the GPU version of the earthquake engine, namely SEM3D-GPU to improve this tool-chain coupling multiple softwares, in the occasion of the abovementioned SMATCH international benchmark. The allocation will help finalizing the set of validation benchmarks (started during the 2024 IDRIS Hackathon) on the new GPU version of the code and to assess the feasibility of using it as a valid and more efficient alternative to its CPU version. Improving AI tools for earthquake generator at super-resolution MUST-ALICE training was performed in supervised way, by adopting the 3-components seismic records (x(t),y(t))belonging STEAD dataset [40]. Since we did not have the possibility of running a simulation per each recorded sample, we low-pass-filtered the latter, namely y(t)to 1 Hz, to obtain x(t). The underlying assumption is that PBS resemble to a filtered version of the recorded time histories. In the meanwhile, we also created two large datasets (HEMEW-3D and HEMEWS-3D, i.e. SEM3D-CPU simulations, accurate up to 5 Hz, considering randomly heterogeneous geological domains and with randomly chosen source point positions and characteristics) to be used as purely synthetic data x(t)and to test MUST-ALICE framework in a blind predictive scheme. Therefore, the second objective of this proposal is to finally be able to validate MUST-ALICE by extending HEMEW-3D and HEMEWS-3D to a larger frequency range and to even more realistic geological profiles and source types by adopting SEM3D-GPU, whose initial performances are listed below. 4 M´ethode 4.1 M´ethode num´erique et impl´ementations (CPU version SEM3D-CPU) We actively develop (jointly with CEA and Insitut de Physique du Globe de Paris) an efficient multi-tool computational tool, called SEM3D-CPU [1, 55], capable of simulating broad-band non6 linear seismic wave propagation in highly heterogeneous media, from the fault to the aboveground structures. [41] compiled a table summarizing major large-scale time-domain simulations of seismic wave motion. Table 2 partially reports the mentioned table and it integrates the table compiled by [13] to show a panorama of the most significant earthquake simulations performed in the last 10 years. The simulation indicated as A7 in Table 2 corresponds to an earthquake sceGrid Size Size f max Ref. Year Method Ressources DOFs (m) (km×km×km) (Hz) Topography [42] 2010 FDM 6 CPUs - 25/125 - 2.5 no 2010 SEM 32 CPUs 66187872 150 - 2.0 yes 2010 SEM 63 CPUs 39902676 20-900 - 3.0 yes 2010 dG 510 CPUs - 200-5000 - 3.0 yes [43] 2010 FEM - 251457147 var 600×300×80 0.5 no 2010 FDM - 2.355 billions 200 500×250×50 0.5 no 2010 FDM - 5.419 billions 100 600×300×80 0.5 no [44] 2010 SEM 192 GPU 131000256 - chunk of earth 0.7 no [45] 2010 FDM 1308 billions 40 810×405×85 2.0 no [46] 2012 SEM 896 GPU 22 billions 24000 Western Europe×200 0.125 yes [7] 2013 FEM 24000 CPUs 15.9 billions 5.5 88 180×135×32 4.0 no [47] 2014 dG 1400832 CPUs 96 billions - - 10 yes [48] 2017 FEM 294912 CPUs 10.7 billions 0.66 2 ×2×0.1 - yes [13] 2017 FDM 1014000 CPUs 23.4 trillions 8 320×312×40 18 yes [21] 2020 FDM 131072 CPUs 8.88 billions 8 100×40×30 5 no A7 2020 SEM 4000 CPUs 13.5 billions 35 130 44 ×44 ×63 10 no TABLE 2 – Summary of major time-domain, large-scale simulations of seismic waves in the last 10 years (table after [41]). FDM : Finite Different Method, FEM : Finite Element Method, SEM : Spectral Element Method, dG : discontinuous Galerkin. ”-” indicates that the desired data is not available in the reference. nario of the Argostoli experimental site, performed on Occigen during allocation A7 [27, 49]. As a matter of fact, Table 2 highlights the fact that SEM3D-CPU represents a competitive alternative to modern broad-band synthetic earthquake simulators world-wide, considering the valuable accuracy relatively to the limited computational resources employed. The core of the platform is represented by a Spectral Element (SE) Method code, tailored to solve 3-D wave propagation in anisotropic, randomly heterogeneous and non-linear geo-materials. The experimental design (ED) foreseen for this project is constructed by SEM3D-CPU analyses. However, the platform is featured by a series of other computational tools that help in constructing large scale numerical scenario of an earthquake event, namely : —HexMesh 5: a scalable octree-based mesh generation code; —randomField 6: a scalable random field generator over large computational domains, based on the spectral generation technique [50]. The parallel octree-based mesh generator is able to treat immersed geometries with high scalability in executions. It was tested up to 6561 processing CPUs. The mesh can be non-conforming, composed exclusively by hexahedrons, or it can be made conforming using tetrahedral elements. Non-conforming meshes have hanging nodes, (i.e. nodes located inside edges or faces), which restricts its use to numerical methods that provide support for this type of nodal interpolation. In particular, efficient parallel implementation of the SEM requires the use of conformal hexahedra. Owing to the SEM spectral convergence, higher resolution is obtained by p-refinement (i.e. by increasing the polynomial order) rather than by increasing the number of linear elements (common practice in Finite Difference/Element schemes). The random field generator employs the sum of a series of Nϕcosines with random phases and random amplitudes [51], sampled over fine regular grids [23]. The Fast Fourier Transform can be used to bring the complexity of this generation method to O(Nϕlog Nϕ). The field is then reinterpolated over the not uniformly space-distributed SE grid (featured by 5 to 10 Gauss-LobattoLegendre points per dimension). When dealing with large domains, the scalability issue is solved 5. https://github.com/jcamata/HexMesh.git 6. https://github.com/cottereau/randomField.git 7 by generating smaller independent realizations, supported on overlapping subdomains (in a distributed memory parallel scheme), and then recomposed together into the full domain. Finally, the non-linear wave-propagation solver exploits the high resolution of the spectral element method with efficient explicit algorithm to integrate non-linear rheology representing the non-linear cyclic behaviour of soils. In the following, the peculiar features of the mentioned computational platform are described. 4.1.1 Octree-based Mesh Generation : HexMesh Octrees are hierarchical data structures that allows the decomposition a three-dimensional space in regular cubes, called octants. The initial octant normally surrounds all the input domain and is divided in 8 equally spaced cubes - the child octants. Each of these children could be subdivided again, in a recursive process that is usually limited by some criteria (i.e. maximum number of subdivisions, minimum octant size). An octree can be implemented by a tree structure or by a linear octree, where only octants with no children are stored. In this last representation, as also known by linear octree, each leaf is stored as a unique integer - called locational code - which identifies its position and depth level in the tree. Among the benefits of linear octrees is the implicit representation of intermediary octants (non-leaves), making it memory efficient, and better performance on sequential access. Additionally, by not making use of memory pointers, parallel implementation of the linear octree has a lower overhead in interprocess communication (in comparison to other tree structures [52]). We have implemented a set of algorithms that address the special needs of parallel octree meshing. These algorithms are : (i) Octree initialization algorithm that guarantees that all processes have at least one piece of the linear octree. (ii) Refinement algorithm that decompose the octants according to some refinement criterion (i.e. maximum size or position relative to the geometry boundaries), and 2 :1 balancing algorithm that ensure that no neighbor octants differentiate in more than one level [52]. This last algorithm is important because a balanced octree results in more smooth transitions between elements, a feature that has a direct influence on the final mesh quality. It also guarantees that the resulting mesh will not have more than one hanging node for each edge or face, reducing the complexity of conforming mesh process. The resulting code has been written in plain C and makes use of MPI only. The code runs on any standard cluster. A previous version of the code (generating non-conformal hexahedra and conformal tetrahedra meshes) has been tested at Stampede Dell cluster (at TACC) and on a SGI ICE 8400 and on a Oracle cluster in Brazil. Table 3 presents a weak scalability test performed on the mesh of the Kefalonia region (Greece). The communication time (last column) attains a fairly constant percentage of the total time. The code has run over 6561 MPI processes. Cores Levels Nodes Elements Time (s) Comm(%) 81 7 85,602,744 83,102,679 12.061 21 % 243 8 769,790,232 747,937,476 32.994 20 % 729 9 6,926,153,724 6,731,438,013 105.188 25 % 6561 10 62,330,385,168 60,583,119,264 123.066 29 % TABLE 3 – Weak scaling of the mesh generation for the Kefalonia region in Greece. Cores : number of MPI processes; Levels : octree refinement level ; Nodes : number of mesh nodes; Elements : number of mesh elements; Time : generation time; Comm : percentage of the total elapsed time for communication operation. Figure 4 show an example of a mesh generated by HexMesh. 8 (a) (b) FIGURE 4 – Mesh of the Cadarache region [29]. 4.1.2 Random Field Generation : randomField The heterogeneous properties of the Earth’s crust are included in our modelling strategy for large scale earthquake scenarios. In this context, by large we refer to a domain size Lmuch larger than both the correlation length ℓC(or some characteristic size of the fluctuation) and the discretization step h. It is therefore advantageous to represent those fluctuations a scalar random fields (for instance, representing the spatial distribution of the shear-modulus). Those large random fields can be effectively sampled over a coarse grid (with a step size relevant for the correlation length) and then interpolated onto the mesh of interest (provided by the GLL (Gauss-Lobatto-Legendre) grid used in SEM3D-CPU). If the discretization step is much larger than the correlation length, the sampling becomes simple and numerically inexpensive. Indeed, for the mesh considered, the random field is essentially a white noise with Gaussian first-order marginal density [53]. The first-order marginal density is modified locally by combining a direct and inverse Rosenblatt transforms [54]. randomField uses the spectral quadrature formula proposed by [51] to generate the random field, taking advantage of the Fast Fourier Transform algorithms (using the library FFTW 7). The code is written in Fortan95 and exploits HDF5 data structures for I/O operations. To efficiently reduce the generation time for large L/ℓCvalues, randomField is implemented in a highly-scalable localized fashion, exploiting MPI. Independent realizations are generated over subdomains, that are then recomposed into the whole domain, by smoothing the discontinuities at the boundaries between partitions to preserve the correlation structure. Table 4 lists the results of the scalability test performed on randomField, with the standard and localized approaches respectively (i.e. by generating a unique random field over the entire domain (standard approach) or by generating independent realizations over subdomains (localized approach). This scalability test was performed by running the generation code on Occigen cluster. 4.1.3 Non-linear wave propagation : SEM3D-CPU SEM3D-CPU [1, 55] is mainly conceived for applications in geophysics, with the aim of considering realistic 3-D complex geo-structures. It is written in Fortran 95, and uses MPI. The I/O is performed using parallel HDF5 libraries. The meshes are non-structured and composed of conformal hexahedra by employing METIS 8library. The discretization in space is based on spectral elements in space and an explicit scheme in time (both for elastic and non-linear case). The spectral element 7. http://fftw.org/ 8. http://glaros.dtc.umn.edu/gkhome/views/metis 9 files saved are in HDF5 format. The amount of file generated NFILES includes the protections, that are immediately erased after the run, and the mesh files, which are the same for each simulation. In this project, we estimate a total amount of 3 O5-like simulation (for the SMATCH benchmark) of 5400 mono-GPU equivalent hours each and 2000 simulations (to enlarge the datasets for deeplearning) of 6.8 mono-GPU equivalent hours each, for a total of 29800. We aim at performing each simulation over a minimum 40 GPUs A100 (on 5 GPU nodes) and corresponding 320 CPUs cores. This choice justify the request of 30000 scalar mono-GPU equivalent hours on A100 GPU nodes (with 200 hours of trial& error setup, to complete the development, as stated in Section 4.1.4). 5 Plan de gestion de donn´ees (DMP) In terms of memory, we need to save approximately 500 Gb of the actual 3500 Gb demanded per run (we erase the protections). Therefore we estimated 15 Tb maximum of hard-disk allocation. No permanent storage is required (data will be progressively repatriated). Concerning visualization, we would like to have access to a GPU visualization node, with GPU accelerated paraview software. Contour plots have been effectively rendered by GPU accelerated paraview software up until 4 Tb on Occigen visualization node(Broadwell E5-2690 [email protected], 1 GPUs Nvidia Tesla P100 PCIe 12Go). 6 Bibliographie R´ef´erences [1] CEA and CentraleSup´ elec and IPGP and CNRS. SEM3D Ver 2017.04 Registered at French Agency for Protection of Programs (D´ epˆ ot APP), 2017. [2] Fanny Lehmann, Filippo Gatti, Micha¨ el Bertin, and Didier Clouteau. 3d elastic wave propagation with a factorized fourier neural operator (f-fno). Computer Methods in Applied Mechanics and Engineering, 420 :116718, February 2024. [3] Anthony Andrews and Peter Folger. Nuclear power plant design and seismic safety considerations. Technical report, Congressional Research Service, 2012. [4] K. Yomogida and J. T. Etgen. 3-D wave propagation in the Los Angeles basin for the WhittierNarrows earthquake. Bulletin of the Seismological Society of America, 83(5) :1325–1344, 1993. [5] Seiji Tsuboi, Dimitri Komatitsch, Chen Ji, and Jeroen Tromp. Broadband modeling of the 2002 Denali fault earthquake on the Earth Simulator. Physics of the Earth and Planetary Interiors, 139(3-4) :305–313, 2003. [6] K. Tsuda, T. Hayakawa, T. Uetake, K. Hikima, R. Tokimitsu, H. Nagumo, and Y. Shiba. Modeling 3D Velocity Structure in the Fault Region of the 2007 Niigataken Chuetu-Oki Earthquake with Folding Structure. In 4th IASPEI/IAEE International Symposium-Effects of Surface Geology on Seismic Motion, page 1–11, 2011. [7] R. Taborda and J. Bielak. Ground-Motion Simulation and Validation of the 2008 Chino Hills, California, Earthquake. Bulletin of the Seismological Society of America, 103(1) :131–156, 2013. [8] M. Villani, E. Faccioli, M. Ordaz, and M. Stupazzini. High-Resolution Seismic Hazard Analysis in a Complex Geological Configuration : The Case of the Sulmona Basin in Central Italy. Earthquake Spectra, 3(4) :1801–1824, 2014. 16 [9] R. Paolucci, I. Mazzieri, and C. Smerzini. Anatomy of strong ground motion : near-source records and 3D physics-based numerical simulations of the Mw 6.0 May 29 2012 Po Plain earthquake, Italy. Geophysical Journal International, 203 :2001–2020, 2015. doi : 10.1093/gji/ggv405. [10] Seiji Tsuboi, Kazuto Ando, Takayuki Miyoshi, Daniel Peter, Dimitri Komatitsch, and Jeroen Tromp. A 1.8 trillion degrees-of-freedom, 1.24 petaflops global seismic wave simulation on the K computer. The International Journal of High Performance Computing Applications, 30(4) :411–422, 2016. doi : 10.1177/1094342016632596. [11] Chiara Smerzini, Kyriazis Pitilakis, and Kiana Hashemi. Evaluation of earthquake ground motion and site effects in the Thessaloniki urban area by 3D finite-fault numerical simulations. Bulletin of Earthquake Engineering, 15(3) :787–812, Mar 2017. doi : 10.1007/s10518-0169977-5. [12] Carsten Uphoff, Sebastian Rettenberger, Michael Bader, Elizabeth H. Madden, Thomas Ulrich, Stephanie Wollherr, and Alice-Agnes Gabriel. Extreme Scale Multi-physics Simulations of the Tsunamigenic 2004 Sumatra Megathrust Earthquake. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’17, page 21 :1–21 :16, New York, NY, USA, 2017. ACM. [13] Haohuan Fu, Conghui He, Bingwei Chen, Zekun Yin, Zhenguo Zhang, Wenqiang Zhang, Tingjian Zhang, Wei Xue, Weiguo Liu, Wanwang Yin, Guangwen Yang, and Xiaofei Chen. 18.9Pflopss Nonlinear Earthquake Simulation on Sunway TaihuLight : Enabling Depiction of 18-Hz and 8-meter Scenarios. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’17, page 2 :1–2 :12, New York, NY, USA, 2017. ACM. [14] Carsten Burstedde, Lucas C. Wilcox, and Omar Ghattas. p4est : Scalable Algorithms for Parallel Adaptive Mesh Refinement on Forests of Octrees. SIAM Journal on Scientific Computing, 33(3) :1103–1133, 2011. [15] Kevin R Milner, Bruce E Shaw, Christine A Goulet, Keith B Richards-Dinger, Scott Callaghan, Thomas H Jordan, James H Dieterich, and Edward H Field. Toward Physics-Based Nonergodic PSHA : A Prototype Fully Deterministic Seismic Hazard Model for Southern California. Bulletin of the Seismological Society of America, 2021. [16] Marco Stupazzini, Maria Infantino, Alexander Allmann, and Roberto Paolucci. Physicsbased probabilistic seismic hazard and loss assessment in large urban areas : A simplified application to Istanbul. Earthquake Engineering & Structural Dynamics, 50(1) :99–115, 2021. [17] K. Aki and B. Chouet. Origin of coda waves : Source, attenuation, and scattering effects. Journal of Geophysical Research, 80(23) :3322–3342, 1975. [18] Angkeara Svay, V Perron, Afifa Imtiaz, I Zentner, Regis Cottereau, D Clouteau1, P-Y Bard, F Hollender, and Fernando Lopez-Caballero. Spatial coherency analysis of seismic ground motions from a rock site dense array implemented during the Kefalonia 2014 aftershock sequence. Earthquake Engineering & Structural Dynamics, 46(12) :1895–1917, 2017. eqe.2881. [19] F. Gatti, F. Lopez-Caballero, R. Paolucci, and D. Clouteau. Near-source effects and nonlinear site response at Kashiwazaki-Kariwa Nuclear Power Plant, in the 2007 Chuetsu-Oki earthquake : evidence from surface and downhole records and 1D numerical simulations. Bulletin of Earthquake Engineering, 16(3) :1105–1135, Mar 2017. [20] Adam Thelen, Xiaoge Zhang, Olga Fink, Yan Lu, Sayan Ghosh, Byeng D. Youn, Michael D. Todd, Sankaran Mahadevan, Chao Hu, and Zhen Hu. A comprehensive review of digital twin — part 1 : modeling and twinning enabling technologies. Structural and Multidisciplinary Optimization, 65(12) :354, December 2022. 17 [21] David McCallen, Floriana Petrone, Mamun Miah, Arben Pitarka, Arthur Rodgers, and Norman Abrahamson. EQSIM-A multidisciplinary framework for fault-to-structure earthquake simulations on exascale computers, part II : Regional simulations of building response. Earthquake Spectra, 0(0) :8755293020970980, 0. [22] M. Korres, F. Lopez-Caballero, V. Alves Fernandes, F. Gatti, I. Zentner, F. Voldoire, D. Clouteau, and D. Castro-Cruz. Enhanced seismic response prediction of critical structures via 3d regional scale physics-based earthquake simulation. Journal of Earthquake Engineering, 27(3) :546–574, February 2023. [23] F. Gatti, L. De Carvalho Paludo, A. Svay, F. Lopez-Caballero, R. Cottereau, and D. Clouteau. Investigation of the earthquake ground motion coherence in heterogeneous non-linear soil deposits. Procedia Engineering, 199(Supplement C) :2354–2359, 2017. X International Conference on Structural Dynamics, EURODYN 2017. [24] David Castro-Cruz, Filippo Gatti, and Fernando Lopez-Caballero. Impact of regional geology on the prediction of physics-based earthquake simulations : the prediction of the seismic response at the Kashiwazaki-Kariwa nuclear power plant (Japan). Submitted at Soil Dynamics and Earthquake Engineering, 2021. [25] Fanny Lehmann, Filippo Gatti, Micha¨ el Bertin, and Didier Clouteau. Synthetic ground motions in heterogeneous geologies : the hemew-3d dataset for scientific machine learning. January 2024. [26] Fanny Lehmann, Filippo Gatti, Micha¨ el Bertin, and Didier Clouteau. Machine learning opportunities to conduct high-fidelity earthquake simulations in multi-scale heterogeneous geology. Frontiers in Earth Science, 10 :1029160, November 2022. [27] Sara Touhami, Fernando Lopez-Caballero, and Didier Clouteau. A holistic approach of numerical analysis of the geology effects on ground motion prediction : Argostoli site test. Journal of Seismology, pages 1–26, 2020. [28] F. Lopez-Caballero and S. Touhami. Numerical Simulation of Wave Propagation in Realistic 3-D Basin Model. In 17th World Conference on Earthquake Engineering, 17WCEE. Sendai, Japan - September 27-October 2, 2021, 2021. [29] David Castro-Cruz, Filippo Gatti, and Fernando Lopez-Caballero. High-fidelity broad-band prediction of regional seismic response : a hybrid coupling of Physics-Based synthetic simulation and Empirical Green’s functions. Submitted at Natural Hazards, 2021. [30] Filippo Gatti and Didier Clouteau. Towards blending physics-based numerical simulations and seismic databases using generative adversarial network. Computer Methods in Applied Mechanics and Engineering, 372 :113421, 2020. [31] Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A. Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. (arXiv :1711.11586), October 2018. arXiv :1711.11586 [cs, stat]. [32] M. Kristekova, J. Kristek, and P. Moczo. Time-frequency misfit and goodness-of-fit criteria for quantitative comparison of time signals. Geophysical Journal International, 178(2) :813–825, 2009. doi : 10.1111/j.1365-246X.2009.04177.x. [33] Muneo Hori and Tsuyoshi Ichimura. Application of Macro-Micro Analysis Method to Estimate Strong Motion Distribution and Resulting Structure Response. [34] T Ichimura, M Hori, PE Quinay, MLL Wijerathne, T Suzuki, and S Noguchi. Comprehensive numerical analysis of fault-structure systems–Computation of the large-scale seismic structural response to a given earthquake scenario–. Earthquake Engineering & Structural Dynamics, 41(4) :795–811, 2012. 18 [35] Tsuyoshi Ichimura, Kohei Fujita, Atsushi Yoshiyuki, Pher Errol Quinay, Muneo Hori, and Takashi Sakanoue. Performance Enhancement of Three-Dimensional Soil Structure Model via Optimization for Estimating Seismic Behavior of Buried Pipelines. Journal of Earthquake and Tsunami, 11(05) :1750019, 2017. doi : 10.1142/S1793431117500191. [36] D Castro-Cruz, F Gatti, F Lopez-Caballero, F Hollender, E El-Haber, and M Causse. Blind broad-band (0-10 hz) numerical prediction of the 3-d near field seismic response of an m w6.0 extended fault scenario : application to the nuclear site of cadarache (france). Geophysical Journal International, 232(1) :581–600, October 2022. [37] Jacobo Bielak, Kostas Loukakis, Yoshiaki Hisada, Chiaki Yoshimura, Y Chiaki, Antonio Ferna, Jacobo Bielak, Yoshiaki Hisada, and Antonio Ferna. Domain Reduction Method for Three-Dimensional Earthquake Modeling in Localized Regions. Part 1 : Theory. Bulletin of the Seismological Society of America, 93(2) :817–840, 2003. [38] P. E. B. Quinay, T. Ichimura, M. Hori, A. Nishida, and S. Yoshimura. Seismic Structural Response Estimates of a Fault-Structure System Model with Fine Resolution Using Multiscale Analysis with Parallel Simulation of Seismic-Wave Propagation. Bulletin of the Seismological Society of America, 103(3) :2094–2110, 2013. doi : 10.1785/0120120216. [39] David McCallen, Anders Petersson, Arthur Rodgers, Arben Pitarka, Mamun Miah, Floriana Petrone, Bjorn Sjogreen, Norman Abrahamson, and Houjun Tang. EQSIM-A multidisciplinary framework for fault-to-structure earthquake simulations on exascale computers part I : Computational models and workflow. Earthquake Spectra, 0(0) :8755293020970982, 0. [40] S Mostafa Mousavi, Yixiao Sheng, Weiqiang Zhu, and Gregory C Beroza. Stanford earthquake dataset (stead) : A global data set of seismic signals for ai. IEEE Access, 2019. [41] Babak Poursartip, Arash Fathi, and John L Tassoulas. Large-scale simulation of seismic wave motion : A review. Soil Dynamics and Earthquake Engineering, 129 :105909, 2020. [42] Emmanuel Chaljub, Peter Moczo, Seiji Tsuno, Pierre-Yves Bard, Jozef Kristek, Martin K¨ aser, Marco Stupazzini, and Miriam Kristekova. Quantitative comparison of four numerical predictions of 3D ground motion in the Grenoble Valley, France. Bulletin of the Seismological Society of America, 100(4) :1427–1455, 08 2010. doi : 10.1785/0120090052. [43] Jacobo Bielak, Robert W. Graves, Kim B. Olsen, Ricardo Taborda, Leonardo RamirezGuzman, Steven M. Day, Geoffrey Ely, Daniel Roten, Thomas Jordan, Philip Maechling, John Urbanic, Yifeng Cui, and Gideon Juve. The ShakeOut earthquake scenario : Verification of three simulation sets. Geophysical Journal International, 180 :375–404, 01 2010. [44] Dimitri Komatitsch, Dominik Goddeke, Gordon Erlebacher, and David Michea. Modeling the propagation of elastic waves using spectral elements on a cluster of 192 GPUs. Computer Science Research and Development, 25(1-2) :75–82, 2010. [45] Yifeng Cui, Kim B Olsen, Thomas H Jordan, Kwangyoon Lee, Jun Zhou, Patrick Small, Daniel Roten, Geoffrey Ely, Dhabaleswar K Panda, Amit Chourasia, et al. Scalable earthquake simulation on petascale supercomputers. In High Performance Computing, Networking, Storage and Analysis (SC), 2010 International Conference for, page 1–20. IEEE, 2010. [46] M. Rietmann, P. Messmer, T. Nissen-Meyer, D. Peter, P. Basini, D. Komatitsch, O. Schenk, J. Tromp, L. Boschi, and D. Giardini. Forward and adjoint simulations of seismic wave propagation on emerging large-scale GPU architectures. In SC ’12 : Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, page 1–11, Nov 2012. doi : 10.1109/SC.2012.59. [47] A. Heinecke, A. Breuer, S. Rettenberger, M. Bader, A. Gabriel, C. Pelties, A. Bode, W. Barth, X. Liao, K. Vaidyanathan, M. Smelyanskiy, and P. Dubey. Petascale High Order Dynamic Rupture Earthquake Simulations on Heterogeneous Supercomputers. In SC ’14 : Proceedings 19 of the International Conference for High Performance Computing, Networking, Storage and Analysis, page 3–14, Nov 2014. doi : 10.1109/SC.2014.6. [48] T. Ichimura, K. Fujita, S. Tanaka, M. Hori, M. Lalith, Y. Shizawa, and H. Kobayashi. PhysicsBased Urban Earthquake Simulation Enhanced by 10.7 BlnDOF x 30 K Time-Step Unstructured FE Non-Linear Seismic Wave Simulation. In SC ’14 : Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, page 15–26, Nov 2014. doi : 10.1109/SC.2014.7. [49] Sara Touhami. Numerical modeling of seismic field and soil interaction : application to the sedimentary basin of Argostoli (Greece). PhD thesis, Th` ese de doctorat en G´ enie civil universit´ e Paris-Saclay 2020, 2020. 2020UPASC007. [50] Masanobu Shinozuka and George Deodatis. Simulation of stochastic processes by spectral representation. Applied Mechanics Reviews, 44(4) :191–204, 1991. [51] M. Shinozuka and G. Deodatis. Simulation of multi-dimensional Gaussian stochastic fields by spectral representation. Applied Mechanics Reviews, 49 :29–53, 1996. [52] Hari Sundar, Rahul S Sampath, and George Biros. Bottom-up construction and 2 : 1 balance refinement of linear octrees in parallel. SIAM Journal on Scientific Computing, 30(5) :2675–2708, 2008. [53] L Paludo, V Bouvier, R Cottereau, and D Clouteau. Efficient Parallel Generation of Random Field of Mechanical Properties for Geophysical Application. In 6th International Conference on Earthquake Geotechnical Engineering, 2015. [54] Murray Rosenblatt. Remarks on a multivariate transformation. The annals of mathematical statistics, 23(3) :470–472, 1952. [55] Sara Touhami, Filippo Gatti, Fernando Lopez-Caballero, R´ egis Cottereau, L´ ucio de Abreu Corrˆ ea, Ludovic Aubry, and Didier Clouteau. Sem3d : A 3d high-fidelity numerical earthquake simulator for broadband (0–10 hz) seismic response prediction at a regional scale. Geosciences, 12(3) :112, March 2022. [56] S. Khazaie, R Cottereau, and D Clouteau. Numerical observation of the equipartition regime in a 3D random elastic medium, and discussion of the limiting parameters. Computers & Geosciences, 102 :56–67, 2017. [57] Dimitri Komatitsch, Seiji Tsuboi, Chen Ji, and Jeroen Tromp. A 14.6 Billion Degrees of Freedom, 5 Teraflops, 2.5 Terabyte Earthquake Simulation on the Earth Simulator. In Proceedings of the 2003 ACM/IEEE Conference on Supercomputing, SC ’03, page 4–, New York, NY, USA, 2003. ACM. [58] P. Cupillard, E. Delavaud, G. Burgos, G. Festa, J.-P. Vilotte, Y. Capdeville, and J.-P. Montagner. RegSEM : a versatile code based on the spectral element method to compute seismic wave propagation at the regional scale. Geophysical Journal International, 188(3) :1203–1220, 2012. doi : 10.1111/j.1365-246X.2011.05311.x. [59] G. Festa and J.-P. Vilotte. The Newmark scheme as velocity-stress time-staggering : an efficient PML implementation for spectral element simulations of elastodynamics. Geophysical Journal International, 161(3) :789–812, 2005. doi : 10.1111/j.1365-246X.2005.02601.x. [60] B. Lombard and J. Piraux. Numerical modeling of transient two-dimensional viscoelastic waves. Journal of Computational Physics, 230(15) :6099–6114, 2011. doi : 10.1016/j.jcp.2011.04.015. [61] Emmanuel Chaljub, Emeline Maufroy, Peter Moczo, Jozef Kristek, Fabrice Hollender, PierreYves Bard, Enrico Priolo, Peter Klin, Florent De Martin, Zhenguo Zhang, et al. 3-D numerical simulations of earthquake ground motion in sedimentary basins : testing accuracy through stringent models. Geophysical Journal International, 201(1) :90–111, 2015. doi : 10.1093/gji/ggu472. 20 [62] Sara Touhami, F Gatti, Fernando Lopez-Caballero, Fabrice HOLLENDER, and Edward Marc Cushing. Argostoli site, from site test to numerical model : a holistic approach. In Best Practices in Physics-based Fault Rupture Models for Seismic Hazard Assessment of Nuclear Installations : issues and challenges towards full Seismic Risk Analysis, Cadarache, France, May 2018. 21