BRIDGING THE PERFORMANCE GAP IN EXASCALE ARCHITECTURES: MONTE CARLO APPROACHES TO HIGH-DIMENSIONAL PROBLEMS

247 International Journal of Social and Educational Innovation Vol. 12, Issue 24, 2025 ISSN (print): 2392 – 6252 eISSN (online): 2393 – 0373 DOI: 10.5281/zenodo.17609785 BRIDGING THE PERFORMANCE GAP IN EXASCALE ARCHITECTURES: MONTE CARLO APPROACHES TO HIGH-DIMENSIONAL PROBLEMS Adedeji Daniel GBADEBO Walter Sisulu University, Mthatha, South Africa [email protected] Abstract The paper uses Monte Carlo algorithms applications to solve large-scale and sparse linear systems that have a significant spectrum of application in the modern field of computation science. Monte Carlo techniques can be used to compute accurate approximations by building up the expected value as the trajectory of random walks and the techniques naturally lend themselves to parallel computation by splitting the trajectory among several processors. The necessity of scalable and memory and parallelism numerical approaches has become obvious as the high-performance and exascale computing architectures develop to a new level. The Neumann-Ulam stochastic methods may indeed offer a valid alternative to the traditional solvers, including direct and iterative ones. This research article uses Python-program-based applications to analyse the speed of computation on the systems of varying size (n = 100, 500 and 1000). The data indicates that the method would perform comparable in terms of accuracy with the more expensive standard direct solvers and that the absolute error can be reduced with the number of samples used. The algorithm has good scaling properties suggesting that it possibly can be efficient in running large-scale scientific and engineering tasks. The results support the general agenda of introducing probabilistic numerical approaches into the computational pipelines, especially in the context of cases where memory restrictions or heavy needs of parallelization are essential factors. Keywords: Monte Carlo approaches, Exascale architectures, Neumann-Ulam approach, Stochastic solvers, Direct iterative solvers. International Journal of Social and Educational Innovation (IJSEIro) Volume 12/ Issue 24/ 2025 248 Introduction This exascale computing can mark a revolutionary point of the history of high-performance computing (HPC). Systems like the currently commissioned Frontier by Oak Ridge National Laboratory in 2022 or made the transition over the exascale threshold, which can enable an unprecedented level of possible computation (DOE, 2022). Such technological advances would enable significant breakthroughs in areas where modeling and simulation commonly approach petascale-scale computations, such as climate dynamics, astrophysical processes, genomics informatics and nuclear processes modeling (PNNL, 2023). However, hardware innovation alone is not the only factor that matters when it comes to the actualization of such prospects. Instead, it can be critically reliant on the development of software algorithms that can effectively cumulate with complexity of the contemporary exascale systems (TechUK, 2024). The software development remains a severe limitation despite the fact that the hardware frontier has gone far. Traditional algorithm-based solutions might not be suitable when scaled to the exascale range where the constraints on trial and error, in inter-processor communication, excessive synchronization costs, and poor memory access patterns usually play a limiting role (Zhou et al., 2021). Additionally, the heterogeneity of architectures used in present supercomputing systems also creates extra complexities, thus requiring further differentiated algorithms (Gupta & Bhatia, 2023). Getting to sustainable performance on a large scale might thus depend on an all-encompassing strategy to scalability, including architectural alignment, optimization of the software layers, and mathematical reformulations (Jin et al., 2020). Monte Carlo methods seem especially appropriate among the types of algorithmic paradigm studied to solve such problems. These stochastic algorithms are inherently compatible with parallel framework, and they could be robust when workloads are uneven and their latency is high (Li & Pavlov, 2021). Owing to their probabilistic natures, they are designed to tackle uncertainty and high-dimensional spaces, which is common in fields like stochastic modeling, risk quantification, and statistical mechanics (Wang et al., 2022). The recent empirical studies have proposed that Monte Carlo algorithm in properly tuned conditions can scale to thousands of processing units in near-linear time (Zhang & Matsuoka, 2023). The article describes the basics of the design of scalable Monte Carlo algorithms to solve large sparse linear systems, and especially the Neumann-Ulam formulation. Our extensions to computational finance (another area that is widely modeling-stochastic and computationally challenging, beyond the linear algebra setting) are also beyond the subject of the paper. Monte Carlo methods can possibly provide a desirable alternative to the traditional solvers in the International Journal of Social and Educational Innovation (IJSEIro) Volume 12/ Issue 24/ 2025 249 exascale setting, specifically where decreased synchronization, increased memory costefficiency, and the use of parallel independence are determinants (Kaur et al., 2024). The potential areas of applicability of the exemplified approach include financial applications, e.g., derivative pricing and portfolio risk evaluation, which can specifically take advantage of the features of the probabilistic structure and parallelism of the discussed methods (Cheng et al., 2021). The study relates the theory and simulation validation by attaching the feasibility of Monte Carlo solvers to approximate the linear systems with greater orders as time goes on despite limited computational points. It is work in the context where the effort to alleviate the performance bottlenecks that algorithmic frameworks are facing in the exascale era is ongoing. With the direction of computing infrastructure toward tens of millions and more simultaneous threads, it is possible that the creation of fault-tolerant, architecture-aware algorithms will be necessary. The goal of the study is not only to improve the numerical performance of Monte Carlo methods in the linear-algebra systems and financial computation but also seeks to gain an interdisciplinary approach to proposing a scalable Monte Carlo method in scientific computing. The latter results are of particular importance when advocating the necessity of crosscutting research, which can be implemented across these fields: the field of numerical analysis, the field of high-performance software engineering, and the field of applying domainspecific modeling in the era of exascale computing (Huang et al., 2025). 2. Monte Carlo Algorithm for Solving Linear Systems The issue of solving large-scale linear systems is one of the primary problems of computational science and engineering. The problem is that classical deterministic solvers are resourceintensive to a point where they cannot be used at all, especially when it comes to the systems whose characteristics include either high dimensionality or high sparsity. Such traditional approaches usually face the hurdle of scalability of the memory capacity as well as computational cost as more systems are added. As an alternative possibility, Monte Carlo methods provide a probabilistic model of calculation that can potentially more effectively resolve these problems particularly when distributed or exascale computer architectures are being used. The Monte Carlo techniques seem to be especially applicable to the issues that arise in the field of radiation transport, financial risk modeling, partial differential equation simulation on a large scale (Mascagni & Bailey, 2020). They are stochastic by nature, which helps in load balancing International Journal of Social and Educational Innovation (IJSEIro) Volume 12/ Issue 24/ 2025 250 and being executed in parallel, which may be beneficial in large heterogeneous computing involving large numbers. Additionally, the versatility on the various problem form structure and hardware architectures could be a source of their increasing utility in high-performance computing settings. With the growing availability of exascale systems, Monte Carlo solvers may have the potential of assisting scientific research at new levels of scale and at a significant level with an alternate to the more stiff deterministic solvers. The core idea of Monte Carlo algorithms for linear systems is to estimate the solution 𝑥 ∈ ℝ𝑛 to the equation 𝐴𝑥 = 𝑏, where 𝐴 ∈ ℝ𝑛×𝑛, using a stochastic process. Specifically, under suitable conditions, the inverse of 𝐴 can be represented by a Neumann series. Suppose that the system is preconditioned or transformed such that 𝐴 = 𝐼 − 𝑀, with 𝜌(𝑀)< 1, where 𝜌(⋅) denotes the spectral radius. The inverse of 𝐴 can then be expressed as: 𝐴−1 =(𝐼 − 𝑀)−1 =∑𝑀𝑘 ∞ 𝑘=0 , (1) which yields the formal solution: 𝑥 = ∑𝑀𝑘 ∞ 𝑘=0 𝑏. (2) The Monte Carlo algorithm applied to linear algebra, and specifically designed for solving large linear systems is provided. This method is valuable when dealing with sparse or very large matrices, and is particularly well-suited for parallel and exascale computing environments due to its probabilistic and iterative nature. Algorithm Steps (Monte Carlo Neumann-Ulam Method) 1. Initialization o Choose the number of histories 𝑁 (number of random walks). o Choose a preconditioner or transform 𝑀 such that 𝐴 = 𝐼 − 𝑀 and 𝜌(𝑀)< 1. 2. Random Walk Generation For each 𝑖 = 1, …, 𝑁: o Start at a randomly selected row 𝑟0. o For each step 𝑘, select the next state 𝑟𝑘+1 based on transition probabilities derived from 𝑀 (e.g., normalize row weights). o Accumulate the contribution of each walk to the estimate of 𝑥 using weights from 𝑏 and matrix entries. 3. Estimate Solution o The result 𝑥𝑗 at each index 𝑗 is computed as the mean contribution of all walks ending at or passing through 𝑗. International Journal of Social and Educational Innovation (IJSEIro) Volume 12/ Issue 24/ 2025 251 Some Key Features • Parallelizable: Each random walk is independent and can be executed in parallel (ideal for GPU and exascale architectures). • Scalable: Handles sparse matrices well, with memory and computation growing linearly with matrix size. • Probabilistic Accuracy: The solution is approximate and improves with more walks 𝑁. 3. Results Each term in the corresponding matrix expansion may be interpreted as a weighted contribution derived from successive matrix-vector multiplications, which can be approximated stochastically through the use of random walks. This strategy forms the basis of the NeumannUlam Monte Carlo method, wherein an ensemble of random walks is generated to probabilistically estimate the contributions to the solution vector 𝐱 (Ji et al., 2022). The general structure of the Monte Carlo solver typically begins with the derivation of a transition probability matrix 𝐏, constructed from the original matrix 𝐌, often through row normalization of its nonzero elements. They then initialise a collection of statistically independent, random walks in which each walk follows a chain of transitions according to the probability structure encoded in 𝐏., and along the way a scalar weight is acquired by adding together the initial right-hand side vector 𝐛 and the matrix entries visited. The estimate 𝑥𝑗 for the 𝑗-th component of the solution of all the random walks which either end at state j, or pass through it, the estimate 𝑥𝑗 is then reached. In the case of finite variance and appropriate conditioning of the matrix 𝐌,, the estimator does not show bias and can approach the actual answer with more walks (Ubaru, Chen, & Saad, 2017). There are a number of computational advantages brought by this stochastic formulation. Remarkably, it is tolerant to parallelism, by the fact that random walks could be performed in parallel, that attribute which makes the method so acceptable by massively parallel hardware like GPU clusters and exascale computing frameworks. Also, the algorithm has low memory requirements and does not require matrix factorization or significant intermediate storage. Monte Carlo solvers can also exhibit good behaviour with sparse, or diagonally dominant matrices, situations in which the standard iterative schemes can be problematic due to the lack of convergence or may require advanced preconditioning strategies (Mascagni & Bailey, 2020; Huang et al., 2025). International Journal of Social and Educational Innovation (IJSEIro) Volume 12/ Issue 24/ 2025 252 Monte Carlo solvers are applicable in a range of fields in science and engineering. As an example, neutron transport equations are problems involving inherently probabilistic particle interactions, and random walk models are very applicable. These solvers are applicable in computational finance to discretize and solve partial differential equations concerned with the option price or to model risk propagation through a large scale financial system. They can also be adapted to resilient, fault-tolerant computing environments operating at scale because of their tolerance of localized error and ability to provide asynchronous updates (Rahman & Liu, 2023). However, it is not free of disadvantages of such a class of algorithms. This can specifically occur in the case of high variance in the estimator or eigenvalues of the matrix spectrum being close to unity. In order to alleviate these shortcomings, importance sampling, variance reduction and stratified sampling have been suggested to enhance the effectiveness of computation. Hybrid frameworks which mix deterministic preconditioners with Monte Carlo estimators have also been recently developed, and they could achieve good trade-offs between accuracy and scalability (Zhang & Matsuoka, 2023). 3.1. Main Outcome Figure 1a presents a comparative visualization of the solution vectors obtained via the Monte Carlo method and a standard direct solver. The analysis considers a sparse linear system of dimension n=100n = 100n=100, with the deterministic reference computed using the spsolve routine. The left subplot displays the element-wise solution, where the solid curve represents the output from the direct solver—serving as a benchmark—while the dashed curve corresponds to the stochastic estimate derived through the Monte Carlo approach. The close alignment between the two curves suggests that the Monte Carlo method can produce an approximate solution that reasonably reflects the deterministic benchmark. This observation holds despite the relatively modest configuration of 2,000 random walks and a maximum walk length of 50, indicating that even limited sampling may suffice for small-scale systems under certain conditions. Figure 1b depicts the absolute error associated with each index, computed as the pointwise difference between the Monte Carlo estimate and the direct solver result. Although minor variations are visible across indices, the errors generally remain bounded and modest in magnitude. The absence of significant outliers or large deviations may indicate a controlled variance in the stochastic estimator under the current simulation parameters. These error levels, International Journal of Social and Educational Innovation (IJSEIro) Volume 12/ Issue 24/ 2025 253 while not negligible, appear acceptable given the scale of the problem and the simplicity of the Monte Carlo configuration employed. Such empirical findings are broadly consistent with theoretical expectations for Monte Carlobased solvers applied to linear systems. Specifically, the accuracy of the solution is expected to improve with an increasing number of random walks, albeit at the expense of higher computational cost (Mascagni & Bailey, 2020; Ji et al., 2022). The demonstrated performance reinforces the potential viability of Monte Carlo methods as alternatives to conventional deterministic solvers, particularly for large and sparse systems where scalability becomes a limiting factor. Furthermore, the inherently parallel structure of Monte Carlo algorithms may make them particularly advantageous in exascale computing environments, where abundant computational resources can be harnessed to increase both precision and efficiency without incurring significant synchronization overhead. Figure 1a: Solution Comparism Direct Solve & Monte Carlos Figure. 1b: Absolute Error per Index Source: Author (2025) 3.2. Sensitivity Analysis To assess the robustness and scalability of the Monte Carlo solver, a sensitivity analysis was conducted by extending the implementation to larger system sizes, specifically n=500n = 500n=500 and n=1000n = 1000n=1000. The findings, which are represented in Figures 2 and 3, show empirical evidence regarding the ability of the algorithm to scale and maintain accuracy in solving big and sparse linear systems. In both figures, the plots indicate that the Monte Carlo estimates are closely clustered with the reference solutions, which are produced by the deterministic sparse solver (spsolve), and therefore, the stochastic method could still yield meaningful estimates with a rise in the dimension of the systems. International Journal of Social and Educational Innovation (IJSEIro) Volume 12/ Issue 24/ 2025 254 In the n = 500n = 500n = 500 case, Figure 2 in the left-hand panel, there is a good correspondence between stochastic estimate and direct solution with only small deviations at isolated indices occurring. The adjacent absolute error plot on the right also shows that the said discrepancies are quite small and equally spread throughout the solution vector. This can be an indication that the algorithm is numerically stable and good at moderate problem sizes, especially when the same set up is used (2000 random walks and walk length of 50 is kept intact). Such observations are aligned with both theoretical properties of Monte Carlo methods that generally enjoy the benefit of statistical averaging over independent trajectories and positive convergence behavior in well-conditioned situations. As system size grows, the preservation of the low and bounded profiles of the errors of the various components of vectors could be a feature that indicates the preservation of the viability of the method. This is of extreme relevance in the sphere of large-scale simulations where the traditional solvers are usually limited in the sphere of memory or communication overhead. The results indicate that use of minimal tuning of the Monte Carlo approach is able to support the problem increase in size without significant loss in the quality of solution. Figure 2: Solution Comparism for Direct Solve and Monte Carlos (LHS) and Absolute Error per Index (RHS) Source: Author (2025) The solution generated by the Monte Carlo method does not seem to get lost as the dimensionality of the system goes to n=1000n = 1000n=1000, with a slight rise in noise figure in the approximation. The associated absolute error diagram also displays the spread of deviations, which can be explained by the fact that stochastic variability combines when International Journal of Social and Educational Innovation (IJSEIro) Volume 12/ Issue 24/ 2025 255 moving to a larger solution space due to sampling. However, the lack of apparent erroneous spikes and the fact that the entire structure of the solution is maintained indicate that the method is persevering with this elevated scale. The results are consistent with theoretical expectations of the convergence of Monte Carlo solvers in mean-square norm and that the method is accurate with the sampling of the number of trajectories (Mascagni & Bailey, 2020; Ji et al., 2022). This result also demonstrates the practical usefulness of Monte Carlo methods in solving problems of high dimension where deterministic applications of such solvers can prove highly impractical either in terms of computational cost or memory requirement. The parallelism of the algorithm is intrinsic to individual random walks, and therefore, the algorithm is quite interesting to run on modern high performance and exascale computers (Ubaru, Chen, & Saad, 2017). The convergence behaviour with respect to the size of the problems thereby implies that the Monte Carlo solvers might be a scalable and resources-effective means of solving large sparse linear systems. Figure 3: Solution Comparism for Direct Solve and Monte Carlos (LHS) and Absolute Error per Index (RHS) Source: Author (2025) Conclusions This study considers the scalability and applicability of the Monte Carlo algorithms, especially the Neumann-Ulam approach, to the large, sparse linear systems. The combination of theoretical exposition and empirical simulations has supported the findings that stochastic solvers could produce reliable approximation in scaling with the dimensionality of problems, as well as low computational requirements. The universal parallelism and probabilistic nature of the Monte Carlo methodology makes it one of the prime candidates to be used when direct