Quantifying performance: Benchmarks: v.1.0

Quantifying performance Benchmarks V 1.0 José M. Cámara ([email protected]) Motivation Performance must be measured to: Assess the behavior of a computing system. Compare various systems. Optimize utilization. Remove bottlenecks. Quantifying performance is a daunting task: Systems are distinct from one another. They are very complex. Handle a broad variety of applications and data. Metrics Latency: time to complete an action: deliver a message, execute a program, fulfill a request… According to the scenario it can be expressed in terms of: execution time or response time too. Throughput: tasks completed per time unit: instructions, messages, queries… Throughput = 1/ latency only when no overlap is produced (instruction or message pipe-line). Otherwise throughput > 1/ latency. Benchmarks Concept: application program used to quantify computer’s performance. Goal: results must be numeric, objective and fair. Types:  Real programs. Synthetic. Kernels. Toys. Suites. Benchmarks Real programs may seem the most objective option but in many occasions their results are hard to interpret since too many computer systems are affected by them in uncertain and variable ways. Synthetic benchmarks are designed to reflect the performance of certain subsystems. Therefore, in order to evaluate all subsystems, usually a suite is preferred. Kernels are close to real programs. They eliminate all that is not relevant, such as user interface, calculation results, etc. Toy benchmarks are short programs that produce results already known to the user. Evaluating results Benchmark suites are a common tool but since they are composed by a number of programs. Systems may perform differently under each one. Direct comparison is not possible then. A more elaborated metric must be obtained: Arithmetic mean:𝐴𝐴𝐴𝐴 = ∑ 𝑟𝑟𝑖𝑖 𝑁𝑁 𝑖𝑖 𝑁𝑁. Not accurate when the tests are unrelated. The longest test tends to prevail. Geometric mean:𝐺𝐺𝐴𝐴 = ∏ 𝑟𝑟𝑖𝑖 𝑁𝑁 𝑖𝑖 𝑁𝑁 . Its utility is unclear. Harmonic mean:𝐻𝐻𝐴𝐴 =𝑁𝑁 ∑ 1 𝑟𝑟𝑖𝑖 𝑁𝑁 𝑖𝑖 Results 𝑟𝑟𝑖𝑖 may be absolute execution times (or their inverses) or speed-ups (referred to a precise system). Comparing means Let’s think of a set of tests where the execution times are: 1s, 4s and 10s. The arithmetic mean of them is 5s. This is correct but, taking into account that the shortest programs may be executed more times than the longest, maybe they should weigh more on the average. The harmonic mean is 3/1,35 = 2,22s. That could reflect more accurately the expected performance of the system. Selecting benchmarks Distinct systems require distinct benchmarks. We are considering two types of servers: supercomputers & data centers. For supercomputers, regardless the subsystem we intend to stress, what matters is execution time (throughput is also relevant in many cases). Results are usually given in terms of FLOPS. For data centers the response time is what the user perceives as “performance”. Throughput doesn’t make much sense in this environment. Results may be given in terms of SLO/SLA ratio. For example: 99% of responses below 100ms. Benchmarking supercomputers By far, the most popular benchmark for supercomputers is Linpack. Provides performance in FLOPS under the execution of a kernel based on the resolution of systems of linear equations. It has a number of variations to adapt itself to many computer architectures. Is the test bed for the Top500 supercomputer rank. NAS NPB: suite of kernels developed by NASA. Kernels try to reflect the core of calculations commonly performed by fluid mechanics applications and other usual programs related to its activity. All benchmarks impose heavy calculation on the system; differ on the problem they solve and the amount of communication traffic they generate: EP: Embarrassingly Parallel. It involves calculation but very little communication between processors. MG: Multigrid. Unlike the former one, it requires communication between both close and remote processors. CG: Conjugate Gradient. Communication is low and scattered among close and remote nodes. FT: Fourier Transform. Heavy communication pattern evenly distributed. IS: Integer Sort. Communication, although heavy and uniform is not as heavy and uniform as in the previous case. LU, SP & BT: address the same mathematical problem using different algorithms. BT is “less parallel” than the rest.