HPC Bytes: Basics of Parallelization with OpenMP

HPC-Bytes: Basics of Parallelization with OpenMP Competence Center for High Performance Computing in Hessen (HKHLR) Dr. Tim Jammer HKHLR is funded by the Hessian Ministry of Sciences and Arts HKHLR Staff HPC-Bytes: Basics of Parallelization with OpenMP 2025-10-16 3/17 What is the HKHLR? Hessisches Kompetenzzentrum für Hochleistungsrechnen (HKHLR) Brain Power Supercharges Computing Power ▶Competence Center for High Performance Computing in Hesse ▶Training ▶Consulting ▶Information ▶Monitoring More info on our webpage: https://www.hkhlr.de/en Participating universities: HKHLR Staff HPC-Bytes: Basics of Parallelization with OpenMP 2025-10-16 4/17 Introduction Hessisches Kompetenzzentrum für Hochleistungsrechnen (HKHLR) ▶Modern CPUs have multiple cores ▶Even more in an HPC cluster ▶HPC hardware is not faster per se but offers many CPU cores ▶Different CPU cores can compute in parallel ▶Distribute the work among several CPU cores HKHLR Staff HPC-Bytes: Basics of Parallelization with OpenMP 2025-10-16 5/17 Shared-memory vs. distributed-memory Hessisches Kompetenzzentrum für Hochleistungsrechnen (HKHLR) 19.09.2025 | Introduction to the Lichtenberg High Performance Computer | 8 Cache Parallelisation (2): Shared and distributed memory Shared Memory (e.g. MT or OpenMP) Distributed Memory (e.g. MPI) Interconnect / Network … MT = Multi-Threading (only „inside“ a node) OpenMP = Open Multi Processing (only „inside“ a node) MPI = Message Passing Interface („inside“and „across“nodes, too) Compute Node Cache … Cache Cache MPI HKHLR Staff HPC-Bytes: Basics of Parallelization with OpenMP 2025-10-16 6/17 OpenMP Hessisches Kompetenzzentrum für Hochleistungsrechnen (HKHLR) ▶Set of compiler directives, library routines, environment variables ▶Specify high-level parallelism in C, C++, and Fortran ▶Relatively easy to parallelize existing code ▶Add pragma directive to control parallelism: ▶#pragma omp parallel in C/C++ ▶!$omp parallel in Fortran ▶Compile with -fopenmp compiler option HKHLR Staff HPC-Bytes: Basics of Parallelization with OpenMP 2025-10-16 7/17 Basic Parallelism Hessisches Kompetenzzentrum für Hochleistungsrechnen (HKHLR) C/C++ 1// Assume: int N; 2// Arrays A[N], B[N], C[N]; 3 4#pragma omp parallel for 5for (int i=0; i < N; i++) { 6A[i] = B[i] + C[i]; 7} Fortran 1! Assume: integer :: N 2! real :: A(N), B(N), C(N) 3 4!$omp parallel do 5do i=1, N 6A(i) =B(i) +C(i) 7end do 8!$omp end parallel do ▶Each thread works on a different part of the array ▶Array parts are automatically assigned by OpenMP ▶Fine-tune with the schedule clause HKHLR Staff HPC-Bytes: Basics of Parallelization with OpenMP 2025-10-16 8/17 Synchronization and Safety Hessisches Kompetenzzentrum für Hochleistungsrechnen (HKHLR) ▶Variables are shared by default ▶If multiple threads use the same variable →data race ▶Example on next slide ▶Prevent by controlling access: ▶private: each thread has its own copy ▶reduction: all threads sum the result at the end ▶Example: #pragma omp parallel for default(none) shared(A) private(B,C) ▶Synchronization ensures correct order: ▶#pragma omp barrier: wait until all threads are ready ▶#pragma omp critical: only one thread can execute this block ▶#pragma omp atomic: make operation atomic HKHLR Staff HPC-Bytes: Basics of Parallelization with OpenMP 2025-10-16 9/17 Data Race Example Hessisches Kompetenzzentrum für Hochleistungsrechnen (HKHLR) C/C++ 1// Assume: int N; 2// double sum = 0.0; 3// double A[N]; 4 5#pragma omp parallel for 6for (int i=0; i < N; i++) { 7sum += A[i]; // <-- Data race! 8} Fix by using a reduction: 1#pragma omp parallel for reduction(+:sum) 2for (int i=0; i < N; i++) { 3sum += A[i];//Safe: threads have a local sum 4}// OpenMP will Sum the Sums of all threads Fortran !$omp parallel do do i=1, N sum =sum +A(i) ! <-- Data race! end do !$omp end parallel do Fix by using a reduction: !$omp parallel do reduction(+:sum) do i=1, N sum =sum +A(i) ! Safe now end do !$omp end parallel do HKHLR Staff HPC-Bytes: Basics of Parallelization with OpenMP 2025-10-16 16/17 Additional Resources Hessisches Kompetenzzentrum für Hochleistungsrechnen (HKHLR) ▶https://www.hkhlr.de/en/events/courses-and-tutorials ▶https://tu-darmstadt.cloud.panopto.eu/Panopto/Pages/Viewer.aspx?id=af359b3afa07-4a28-a695-ae8b00fe96d2&start=10.273728 ▶https://www.openmp.org/resources/ HKHLR Staff HPC-Bytes: Basics of Parallelization with OpenMP 2025-10-16 17/17 Image Sources Hessisches Kompetenzzentrum für Hochleistungsrechnen (HKHLR) ▶Slide 4: CC0 via wikimedia commons (https://commons.wikimedia.org/wiki/File:CPU_clock_speed_and_Core_count_Graph.png) ▶Slide 5: CC BY 4.0 HRZ/HPC TU Darmstadt HKHLR Staff HPC-Bytes: Basics of Parallelization with OpenMP 2025-10-16