Making Software Easy-Build Friendly: Streamlining HPC Software Deployment

Making Software EasyBuild Friendly: Streamlining HPC Software Deployment RSECon25 University of Warwick Dr. Tanmoy Chakraborty Research Software Specialist 11th September 2025 1 Motivation 🔗 HPC software stacks are complex and heterogeneous • Wide variety of compilers, MPI implementations, and libraries • Multiple hardware architectures with different optimization needs • Constant updates to scientific packages create compatibility issues ⏳ Manual build are time-consuming and error prone • Repeated trial-and-error during configuration and compilation • Hard to replicate across multiple systems or for other users • Risk of subtle build error that only appear at runtime 📊 Reproducibility, maintainability and scalability are critical • Essential for reliable, long-term scientific research • Standardized builds ensure consistent performance across clusters • Scalable deployment saves time for both RSEs and end-users 2 Introducing EasyBuild Open-source framework to manage scientific software on HPC systems • Automates the entire build, installation, and environment setup process • Eliminates manual compilation errors through a consistent, scripted workflow • Improves reproducibility with archived logs and saved build recipes Flexible and human-readable build recipes (easyconfigs & easyblocks) • easyconfigs: define software name, version, sources, dependencies, and toolchain in a readable format. • easyblocks: Python classes that capture special build logic for reuse • Enables rapid adaptation to new software versions with minimal edits. Robust dependency handling and modular versioning • Automatically builds required dependencies using the --robot option. • Supports multiple software versions without conflicts via isolated paths • Generates module files for seamless integration with environment modules (e.g., Lmod) EasyBuild is a software build and installation framework 3 Introducing EasyBuild Toolchains in EasyBuild • A predefined combination of compiler, MPI library, and possibly math libraries • Ensures compatibility between software components built with the same stack • Examples: foss (GCC + OpenMPI + OpenBLAS + ScaLAPACK) or intel (ICC + Intel MPI + MKL) Role of Lmod in EasyBuild • A Lua-based environment module system that manages software environments • Supports hierarchical module naming to avoid conflicts and improve usability • Integrates with EasyBuild to automatically generate and manage module files Integration with HPC environments • Facilitates loading/unloading software with a single command (module load pkg) • Allows side-by-side availability of multiple versions of the same software • Enables reproducible setups across different HPC systems using shared recipes EasyBuild is a software build and installation framework 4 What EasyBuild is NOT • EasyBuild is not YABT (Yet Another Build Tool) • It does not try to replace CMake,make,pip etc. • It wraps around those tools and automates installation procedures • EasyBuild does not replace traditional Linux package managers (yum, def, apt,..) • You should still install some software via OS package manager: OpenSSl, Slurm, etc. • EasyBuild is not a magic solution to all your (software installation) problems • You may still run into compiler errors (unless somebody worked around it already) 5 EasyBuild terminology: framework & easyblock framework: • The EasyBuild framework is the core of EasyBuild • Collection of python modules, organised in packages • Implements common functionality for building and installing software • Support for applying patches, running commands, generating module files,.. • Examples: easybuild.toolchains, easybuild.tools, … • Provides the eb command • GitHub repo: https://github.com/easybuilders/easybuild-framework easyblock: • A python module that implements a specific software installation procedure • Generic easyblocks examples: Binary, CMakeMake, ConfigureMake, PythonBundle, RPackage,… • Software specific easyblocks: OpenFOAM, TensorFlow, CP2K, WIEN2k,… • Installation procedure can be controlled via easyconfig parameters • Can use directly from GitHub PR (see --include-easyblocks-from-pr) • GitHub repo: https://github.com/easybuilders/easybuild-easyblocks 6 EasyBuild terminology: easyconfig • Text file that specifies what EasyBuild should install (in Python syntax) • Collection of values for easyconfig parameters (key-value definitions) • Filename generally ends with “.eb” • Specific filename is expected in some contexts • Should match with values for name, version, toolchain, versionsuffix • GitHub repository: https://github.com/easybuilders/easybuild-easyconfigs 7 Installing EasyBuild: requirements and options Requirements: • Linux as OS (CentOS, RHEL, Ubuntu, Debian, SLES,…) • Also supports macOS with limited support available • Python >=3.6 is required for EasyBuild 5.0 • Python standard libraries are required for EasyBuild’s core functionality • Python 3 is highly recommend • An environment module tool (module command) • Default is Lua-based Lmod implementation; highly recommended • Tcl-based implementations are also supported Options: • Installing EasyBuild using a standard Python installation tool • pip install easybuild • Or a variant of thereof (pip3 install --user , using virtualenv etc.) • Installing EasyBuild as a module, with EasyBuild (recommended!) • 3-step “bootstrap” procedure • Development setup • Clone GitHub repositories: easybuilders/easybuild-{framework,easyblocks,easyconfigs} • Update $PATH and $PYTHONPATH environment variables 8 Installing EasyBuild: as a module Step 1: Use pip to obtain a temporary installation of EasyBuild export TMPDIR=/tmp/$USER/easybuild pip3 install --prefix $TMPDIR easybuild # update environment to use this temporary EasyBuild installation export PATH=$TMPDIR/bin:$PATH export PYTHONPATH=$TMPDIR/lib/python3.9/site-packages:$PYTHONPATH # instruct EasyBuild to use python3 command export EB_PYTHON=python3 Step 2: Use EasyBuild to install EasyBuild (as a module) in home directory eb --install-latest-eb-release --prefix $HOME/easybuild # and then clean up the temporary EasyBuild installation rm -r $TMPDIRand Step 3: Load EasyBuild module to use for final installation module use $HOME/easybuild/modules/all module load EasyBuild 9 Module naming schemes: Flat vs hierarchical: examples Blythe Sulis 16 Manual vs EasyBuild: an example 17 EasyBuild GitHub integration: set-up Requirements: • A GitHub account • A GitHub user name • A GitHub token + keyring Python package • git command / GitPython Python package • ssh public key registered on GitHub • Fork of the Easybuild repositories on GitHub • For more details see here: https://docs.easybuild.io/integration-with-github/ 18 EasyBuild GitHub integration: contributing to EasyBuild 19 EasyBuild in a nutshell: A few takeaway messages • Tool to provide a consistent and well performing scientific software stack • Uniform interface for installing scientific software on HPC systems • Saves time by automating tedious, boring and repetitive tasks • Can empower scientific researchers to self-manage their software stack • A platform for collaboration among HPC sites worldwide • Supports fully autonomously installing (scientific) software, dependencies, modules • Highly configurable, easy to extend, easy customisation • Detailed logging, support for using custom module naming schemes • Integrates with various other tools (Lmod, Singularity, Slurm, …) • Actively developed and supported by worldwide HPC community • Various support channels (mailing list, Slack, conference calls) + yearly user meetings 20 Questions? • Website: https://easybuild.io • Documentation: https://docs.easybuild.io • Tutorials: https://easybuild.io/tutorial (adapted from: https://tutorial.easybuild.io/2022-isc22/) • Yearly EasyBuild User Meeting: https://easybuild.io/eum • Getting help: • Mailing list: https://lists.ugent.be/wws/subscribe/easybuild • Slack: https://easybuild.slack.com - https://easybuild.io/join-slack • Bi-weekly conference calls: https://github.com/easybuilders/easybuild/wiki/Conference-calls • List of supported software: https://docs.easybuild.io/version-specific/supported-software/ • GitHub repo: https://github.com/easybuilders • YouTube: https://www.youtube.com/c/easybuilders 21