MaRDIMark: the MaRDI benchmarking framework and its instances
Abstract
Research software plays a pivotal role in the modern science workflows, yet numerical experiments often suffer from limitations in reliability, robustness, and interoperability. Despite the development of novel algorithms, their adoption across diverse domains is hindered by the challenges of integrating them into existing workflows. Furthermore, the absence of standardized benchmarks impedes the transparent and reproducible evaluation of algorithm performance. To address these challenges, we propose a standardized specification for benchmarking and evaluating algorithm performance across various mathematical domains. We demonstrate the implementation of MaRDIMark through specific instances tailored to model order reduction, gas network simulation and linear algebra applications. Designed with extensibility, customizability, and reproducibility in mind, these instances offer a flexible and domain-independent approach to benchmarking, facilitating broader adoption and comparative analysis within and across mathematical communities.Presented at GAMM RSE & RDM Kickoff Meeting, Braunschweig, 04-05 Dec 2025.
Full text
MaRDIMark :the MaRDI benchmarking framework and its instances A. S. Nayak, K.Lund, J. Saak, P. Benner December 5, 2025 GAMM RSE/RDM Kickoff Meeting Supported by:
MaRDI: Mathematical Research Data Initiative The German research data consortium for mathematics. 16 institutions and partners. 2021-2026. Supported by National Research Data Infrastructure (NFDI). A.S. Nayak, K.Lund, J. Saak, P. Benner MaRDIMark benchmarking framework and instances 2/18
Outline 1. Introduction 2. The MarDIMark specification 3. MORB: A MarDIMark instance 4. Outlook and Future Work A.S. Nayak, K.Lund, J. Saak, P. Benner MaRDIMark benchmarking framework and instances 2/18
IntroductionIntroduction
Benchmarking : Introduction Benchmarking is a process of comparing the performance of a system or component using a set of standard measures. Requires a problem definition with a set of data and a set of algorithms. Requires a well-defined set of metrics. Atransparent benchmarking process is essential for: identify areas for improvement, optimize efficiency and effectiveness, ensure compliance with standards. Universal Benchmarker The definition is community-specific and, depends heavily on the context. There is no universal benchmarker! 10−1 101 103 Runtime in s PGS PGS V SCQR10 CGS CGS IRO LS CGS P IRO NUMPY QR SCIPY QR 0 50 100 Cores average 0.2 0.4 0.6 L2 miss ratio 0 0.2 0.4 L2 miss rate 012345678 0 10 20 30 𝑚 𝑛in log10 scale CPI A.S. Nayak, K.Lund, J. Saak, P. Benner MaRDIMark benchmarking framework and instances 3/18
Benchmarking : Introduction Benchmarking is a process of comparing the performance of a system or component using a set of standard measures. Requires a problem definition with a set of data and a set of algorithms. Requires a well-defined set of metrics. Atransparent benchmarking process is essential for: identify areas for improvement, optimize efficiency and effectiveness, ensure compliance with standards. Universal Benchmarker The definition is community-specific and, depends heavily on the context. There is no universal benchmarker! 10−1 101 103 Runtime in s PGS PGS V SCQR10 CGS CGS IRO LS CGS P IRO NUMPY QR SCIPY QR 0 50 100 Cores average 0.2 0.4 0.6 L2 miss ratio 0 0.2 0.4 L2 miss rate 012345678 0 10 20 30 𝑚 𝑛in log10 scale CPI A.S. Nayak, K.Lund, J. Saak, P. Benner MaRDIMark benchmarking framework and instances 3/18
Benchmarking : Problem Constituents Database Curation Define benchmark instance and “concrete algorithm” Determine important searchable attributes and aggregate metadata Automate curation process to avoid human error Choose file-naming schemes and standards that are FAIR and conform to community traditions Community Engagement Ensure proper licensing Encourage researchers to contribute their data, provide feedback, and conform to standards KISS1: Reduce barriers to cooperation by providing workflows, GUIs, easy-to-follow guidelines, etc. Benchmark Framework Identify domain-specific aspects of each module Choose intuitive and informative performance measures Develop platform-independent interfaces 1Keep It Simple, Silly https://w.wiki/JAn A.S. Nayak, K.Lund, J. Saak, P. Benner MaRDIMark benchmarking framework and instances 4/18
Benchmarking : Tools Computer Science §google/benchmark §catchorg/Catch2 §pytest-dev/pytest-benchmark ... Focus on testing and providing detailed computational performance metrics for algorithms. Popular due to their easy-to-use yet feature-rich API. Distributed Computing §FZJ-JSC/JUBE §LLNL/benchpark §GoogleCloudPlatform/PerfKitBenchmarker ... Features include reproducible complicated software configurations across distributed systems, parametric runs, and comprehensive reporting. Reproducibility: Consistency in results across multiple runs. Scalability: Handle different sizes of data and systems. Fairness: Unbiased comparison across different hardwares or software tools. Verification: Confirmable and verifiable results. Ease of use: Simple and intuitive. A.S. Nayak, K.Lund, J. Saak, P. Benner MaRDIMark benchmarking framework and instances 5/18
Benchmarking : Tools Computer Science §google/benchmark §catchorg/Catch2 §pytest-dev/pytest-benchmark ... Focus on testing and providing detailed computational performance metrics for algorithms. Popular due to their easy-to-use yet feature-rich API. Distributed Computing §FZJ-JSC/JUBE §LLNL/benchpark §GoogleCloudPlatform/PerfKitBenchmarker ... Features include reproducible complicated software configurations across distributed systems, parametric runs, and comprehensive reporting. Reproducibility: Consistency in results across multiple runs. Scalability: Handle different sizes of data and systems. Fairness: Unbiased comparison across different hardwares or software tools. Verification: Confirmable and verifiable results. Ease of use: Simple and intuitive. A.S. Nayak, K.Lund, J. Saak, P. Benner MaRDIMark benchmarking framework and instances 5/18
MORB : Software Components MORB examples algorithms driver analyzer explorer toolkits bindings https://modelreduction.org MORWiki A.S. Nayak, K.Lund, J. Saak, P. Benner MaRDIMark benchmarking framework and instances 9/18
MORB : Software Components MORB examples algorithms driver analyzer explorer toolkits bindings examples Matrix data checked in LTISystem class. Can retrieve datasets using morb-fetch. Sorts the dataset based on combination of matrices received. Can also be configured with custom datasets. (ongoing) connect to knowledge graphs. morb-fetch §mardi4nfdi/morb-fetch Easy dataset selection, download, extraction Flexible configuration options and dataset-caching Python package with easy installation Platform independent A.S. Nayak, K.Lund, J. Saak, P. Benner MaRDIMark benchmarking framework and instances 9/18
MORB : Software Components MORB examples algorithms driver analyzer explorer toolkits bindings examples Matrix data checked in LTISystem class. Can retrieve datasets using morb-fetch. Sorts the dataset based on combination of matrices received. Can also be configured with custom datasets. (ongoing) connect to knowledge graphs. morb-fetch §mardi4nfdi/morb-fetch Easy dataset selection, download, extraction Flexible configuration options and dataset-caching Python package with easy installation Platform independent A.S. Nayak, K.Lund, J. Saak, P. Benner MaRDIMark benchmarking framework and instances 9/18
MORB : Software Components MORB examples algorithms driver analyzer explorer toolkits bindings algorithms Currently only implements balancing methods. Balanced Truncation (BT) Positive-real BT Stochastic BT Frequency-limited BT Time-limited BT Frequency-weighted BT Linear Quadratic Gaussian BT holds metadata pointing to a MORWiki entry. (extend) connect to knowledge graphs. concrete algorithm Implementable algorithm with fully specified parameters. Stores identifiable information of a toolkit. Can apply() itself on an example. A.S. Nayak, K.Lund, J. Saak, P. Benner MaRDIMark benchmarking framework and instances 10/18
MORB : Software Components MORB examples algorithms driver analyzer explorer toolkits bindings algorithms Currently only implements balancing methods. Balanced Truncation (BT) Positive-real BT Stochastic BT Frequency-limited BT Time-limited BT Frequency-weighted BT Linear Quadratic Gaussian BT holds metadata pointing to a MORWiki entry. (extend) connect to knowledge graphs. concrete algorithm Implementable algorithm with fully specified parameters. Stores identifiable information of a toolkit. Can apply() itself on an example. A.S. Nayak, K.Lund, J. Saak, P. Benner MaRDIMark benchmarking framework and instances 10/18
MORB : Software Components MORB examples algorithms driver analyzer explorer toolkits bindings toolkits M-M.E.S.S MORLAB pyMOR Octave Control Toolbox MATLAB Control Toolbox ...(user extensible) bindings Octave API MATLAB API Python API Julia API C++ API ...(user extensible) driver Parses the problem description and orchestrates it’s execution. Interfaces to different toolkits. Provides executors which manages bindings. A.S. Nayak, K.Lund, J. Saak, P. Benner MaRDIMark benchmarking framework and instances 11/18
MORB : Software Components MORB examples algorithms driver analyzer explorer toolkits bindings toolkits M-M.E.S.S MORLAB pyMOR Octave Control Toolbox MATLAB Control Toolbox ...(user extensible) bindings Octave API MATLAB API Python API Julia API C++ API ...(user extensible) driver Parses the problem description and orchestrates it’s execution. Interfaces to different toolkits. Provides executors which manages bindings. A.S. Nayak, K.Lund, J. Saak, P. Benner MaRDIMark benchmarking framework and instances 11/18
MORB : Software Components MORB examples algorithms driver analyzer explorer toolkits bindings toolkits M-M.E.S.S MORLAB pyMOR Octave Control Toolbox MATLAB Control Toolbox ...(user extensible) bindings Octave API MATLAB API Python API Julia API C++ API ...(user extensible) driver Parses the problem description and orchestrates it’s execution. Interfaces to different toolkits. Provides executors which manages bindings. A.S. Nayak, K.Lund, J. Saak, P. Benner MaRDIMark benchmarking framework and instances 11/18
MORB : Software Components MORB examples algorithms driver analyzer explorer toolkits bindings analyzer Defines proper measures for accuracy, performance and reliability of the algorithms. Error Norms : L1,L2,H2,H∞, ... Bode Magnitudes Singular values Computation Runtimes (extend) upload to knowledge graph. explorer Plots: Timings, Error plots. T EX Report: autogenerated with specifications, extensible templates. Tectonic-based fallback compiling. Easily configurable. PDF Report: Easily distributed and viewed. A.S. Nayak, K.Lund, J. Saak, P. Benner MaRDIMark benchmarking framework and instances 12/18
MORB : Software Components MORB examples algorithms driver analyzer explorer toolkits bindings analyzer Defines proper measures for accuracy, performance and reliability of the algorithms. Error Norms : L1,L2,H2,H∞, ... Bode Magnitudes Singular values Computation Runtimes (extend) upload to knowledge graph. explorer Plots: Timings, Error plots. T EX Report: autogenerated with specifications, extensible templates. Tectonic-based fallback compiling. Easily configurable. PDF Report: Easily distributed and viewed. A.S. Nayak, K.Lund, J. Saak, P. Benner MaRDIMark benchmarking framework and instances 13/18
Outlook Recommend algorithms and examples from and store results in knowledge graphs Bindings should involve open-interfaces: §MaRDI4NFDI/open-interfaces Improve “confirmable workflows” Ensure synergetic potential with other communities and MaRDI projects Integrate into MaRDI portal Extend MORB specification to PH systems (planned) A.S. Nayak, K.Lund, J. Saak, P. Benner MaRDIMark benchmarking framework and instances 18/18
Thank You! Questions? Reach out: [email protected]