D4.5 HPC API and documentation

LUMI AI Factory Service Center Empowering Europe’s AI Ecosystem D4.5. HPC API and documentation 2 D4.5 HPC API and documentation D4.5. HPC API and documentation 3 Project Title LUMI AI Factory Service Center Project Acronym LUMI-AIF Project Number 101234208 Type of Action HORIZON-JU-RIA Topic HORIZON-JU-EUROHPC-2025-AI-01-IBA-01 Starting Date of Project 01.03.2025 Ending Date of Project 29.02.2028 Duration of the Project 36 months Website lumi-ai-factory.eu Work Package WP4 Task Task 4.4. Development of AI-ready computing environment and HPC API Lead Authors Sebastian von Alfthan (CSC), Juha A. Lehtonen (CSC) Contributors Lukas Prediger (CSC), Vaclav Svaton (IT4I), Juha Hulkkonen (CSC) Peer Reviewers Juho Keränen (CSC), Tomáš Martinovič (IT4I) Version 1.0 Due Date 29.8.2025 Submission Date 27.8.2025 Dissemination level x PU: Public SEN: Sensitive – limited under the conditions of the Grant Agreement EU-RES. Classified Information: RESTREINT UE (Commission Decision 2005/444/EC) EU-CON. Classified Information: CONFIDENTIEL UE (Commission Decision 2005/444/EC) EU-SEC. Classified Information: SECRET UE (Commission Decision 2005/444/EC) D4.5. HPC API and documentation 4 Version History Revision Date Editors Comments 0.1. Lukas Prediger Draft for HPC-API User Stories 0.2. 26.6.2025 Juha Hulkkonen Added MLOps content and some proposals 0.3. 2.7.2025 Juha Hulkkonen Drafted the Introduction, restructured a bit and edited the other chapters 0.4 4.7.2025 Sebastian von Alfthan Added first draft for FirecREST + Kubeflow 0.5 11.7.2025 Juha A. Lehtonen Added first draft for FirecREST solution technical overview. 0.6 21.7.2025 Juha Hulkkonen Re-write the Introduction, cleaning and refining other chapters. 0.7 25.7.2025 Juha Hulkkonen Finalizing the draft 0.8 13.8.2025 Lukas Prediger Changes to overall structure 0.9 21.8.2025 Pauliina Somerkoski Resolved the changes and deleted comments. Small finishig touches. 1.0 27.8.2025 Anna Luoma Final quality check performed by the PMO, sent to official review. Glossary of Terms AI Artificial Intelligence Airflow Data-engineering oriented workflow engine Aitta AI Inference service created in CSC API Application Programming Interface FirecREST An open-source web API to access HPC resources. FirecREST is developed by Swiss National Supercomputing Centre (CSCS). HEAppE HEAppE Middleware (High-End Application Execution Middleware) is an open-source software framework to provides a interface to HPC resources. Developed at IT4Innovations National Supercomputing Center. HPC High Performance Computing Item Description Kubeflow A popular workflow system for machine learning operations LUMI-K OpenShift based container cloud currently in development LUMI-O Ceph based object storage connected to LUMI system D4.5. HPC API and documentation 5 ML Machine Learning MLOps Machine Learning Operations – The operation of machine learning systems using automated tools for training, testing, and deployment of machine learning models. OpenID Connect OpenID Connect (OIDC) is an identity layer built on top of OAuth 2.0 framework. It allows third party applications to verify the identity of the user based on authentication performed by an authorization server. REST API Representational State Transfer Application Programming Interface - a common form of a stateless web API using the HTTP protocol S3 Simple Storage Service protocol to access object storage systems Slurm The Slurm Workload Manager is an open-source job scheduler for Linux based systems, used by many of the world's supercomputers and computer clusters. D4.5. HPC API and documentation 6 Executive Summary This document describes HPC-API development for LUMI AI Factory. An HPC-API (High Performance Computing Application Programming Interface) is used to programmatically manage and interact with HPC environments. It functions for job management, like submitting and cancelling jobs on a supercomputer and querying their status, as well as handling data on the system. The HPC-APIs deployed to LUMI are used by access various software components that are being developed for the LUMI AI Factory. Modern AI (Artificial Intelligence) development often relies on an extensive underlying software environment to support the machine learning workloads running in the system. This document describes the deployment of a new HPC-API for LUMI. The aim is to set up an HPC-API that can be used with MLOps tools and containerized workflows to serve users of LUMI AI Factory in AI development tasks. The technology selected for this HPC-API is FirecREST developed by Swiss National Supercomputing Centre (CSCS). This document also describes HEAppE as an alternative approach. HEAppE, developed by IT4Innovations National Supercomputing Center (IT4I), is an open-source software framework that provides a secure and user-friendly interface to HPC resources. HEAppE is already used as part of some CSC services as well as isolated, dedicated HPC-API for some scientific projects. D4.5. HPC API and documentation 7 Table of Contents 1. Introduction............................................................................................................... 8 1.1 Introduction to LUMI AI Factory MLOps environment 8 2. Requirements for HPC-API ........................................................................................ 10 2.1 User stories for HPC-API usage 10 2.1.1 User of LUMI AI Factory service offerings 10 2.1.2 User of LUMI AI Factory HPC-API 11 2.1.3 Client applications hosted by LUMI AI Factory projects 11 3. Existing HPC-API software solutions .......................................................................... 12 3.1 FirecREST HPC-API 12 3.1.1 Description of FirecREST 12 3.1.2 Architecture 13 3.1.3 Features 13 3.1.4 Authentication and authorization 14 3.1.5 Use case support 14 3.2 HEAppE Middleware 16 3.2.1 Motivation 16 3.2.2 Architecture 17 3.2.3 Features and Security 17 3.2.4 Future work and References 18 4. Design of the LUMI AI Factory HPC-API ..................................................................... 18 4.1 Selection of solution components 18 4.2 Implementation plan 19 4.2.1 Integrating FirecREST with LUMI and CSC infrastructure 19 4.2.2 Kubeflow HPC offloading via FirecREST 19 4.2.2.1 Implementation plan 20 4.2.2.2 Alternative approaches and future work 20 D4.5. HPC API and documentation 8 1. Introduction Many commercial services in the field of AI are based on cloud platforms. Even though the latest containerised microservice solutions are capable and scalable, more parallel computing power is needed for the most demanding data pre-processing, AI training, finetuning and inference tasks. The LUMI AI Factory consortium compute resources consist of access to LUMI and, later, LUMI-AI supercomputer as well as strong cloud platforms. Therefore, we are well positioned to empower cloudbased solutions with considerable HPC capacity. Combining these very different computing paradigms is not straightforward task. Programmatic access from cloud services to HPC is needed to combine these platforms. This Application Programming Interface (API) to be developed and delivered is referred as HPC-API in the following. The HPC-API is an essential component to implement the LUMI AI Factory service portfolio as envisioned, of which an overview is presented in the following section. The HPC-API will allow us to connect to HPC capacity of LUMI and later LUMI-AI supercomputers from our cloud-based services. That connection and the emergent combination of those two computing platforms can enable novel ways to develop and offer AI services. Most of the LUMI AI Factory software environment components will require access to supercomputer resources. There are existing technologies currently available to solve this issue, such as FirecREST and HEAppE which are described in more detail in Section 3. They have different approaches which are targeted to different use-cases and technologies. Key considerations for LUMI AI Factory are the convenience of usage, suitability to our needs, authentication and access key management. To select the right approach, requirements for the HPC-API are considered in Section 2, based on which a plan for the implementation of the HPC-API is presented in Section 4. Typical uses cases are workflows supporting use of LLMs for text pre-processing, like dataset curation. Other considered use cases are LLM fine-tuning and LLM evaluation. The aim is to use the HPC-API to access these ready-made containerized workflows on the LUMI system. HPC-API extends the ways user can utilize the HPC resources from external tools or services on LUMI. 1.1 Introduction to LUMI AI Factory MLOps environment LUMI AI Factory provides an MLOps environment designed around centrally operated core components and served applications, that are supported by a larger selection of recommended frameworks and complete MLOps stacks. Planned core components for LUMI AI Factory MLOps environment are the open-source tools Kubeflow, to orchestrate the machine learning workflows, and MLflow, to track the training and to store the models and training metadata. Our self-developed Aitta AI inference service is complementing the offering and closing the MLOps loop visualized in the Figure 1, by providing the HPC-powered inference service even D4.5. HPC API and documentation 9 for the heaviest generative models. Aitta is currently using a dedicated HEAppE instance for accessing the HPC. Figure 1: MLOps Cycle to cover tools for continuous improvement and delivery of machine learning models In the MLOps context the HPC-API will offer the essential avenue for Kubeflow to access LUMI resources to complement LUMI-K container cloud resources for more heavyweight use cases. Kubeflow is a popular workflow system for machine learning operations, enjoying almost a de-facto standard status currently and providing smooth interoperability with commercial cloud environments. Kubeflow excels in orchestrating complex ML training and deployment pipelines, including preprocessing of data, performing training and running model inference. Kubeflow enables industry users who are used to cloud native tools on services offered by American hyperscalers an easy way to start using LUMI AI Factory services. A key target is to enable Kubeflow to run compute-heavy tasks on LUMIG GPU nodes using the HPC-API, while light tasks can remain on LUMI-K. MLflow is one of the core components extending the possibilities of Kubeflow for managing the machine learning lifecycle. It is already available for users on CSC's HPC systems to track ML training metrics locally. With the external MLflow application, a user can track the metrics outside of the HPC system for wider and longer-term usage. In the LUMI AI Factory MLOps toolkit, MLflow is offered as a stand-alone application on top of CSC container clouds, but also as integral part of Kubeflow to offer a seamless way D4.5. HPC API and documentation 16 Figure 4: Standalone API for CLI or desktop tools 3.2 HEAppE Middleware 3.2.1 Motivation HEAppE Middleware (High-End Application Execution Middleware) is an open-source software framework that provides a secure and user-friendly interface to HPC resources. Designed with modularity, multi-user support, and REST API accessibility, HEAppE enables seamless integration of HPC infrastructure with custom user applications, web portals, and third-party services. Developed in-house at IT4Innovations National Supercomputing Center, HEAppE is an implementation of the HPC-as-aService concept. HPC-as-a-Service is a well-known term in high-performance computing. It enables users to access an HPC infrastructure without a need to buy and manage their own physical servers or data center infrastructure. Through this service, small and medium enterprises (SMEs) can take advantage of the technology without an upfront investment in the hardware. This approach further lowers the entry barrier for users and SMEs who are interested in utilizing massive parallel computers but often do not have the necessary level of expertise in this area. D4.5. HPC API and documentation 17 HEAppE manages and provides information about submitted and running jobs and their data between the client application and the HPC infrastructure. HEAppE is able to submit required computation or simulation on HPC infrastructure, monitor the progress and notify the user should the need arise. It provides necessary functions for job management, monitoring and reporting, user authentication and authorization, file transfer, encryption, and various notification mechanisms. 3.2.2 Architecture HEAppE Middleware follows a modular architecture (Figure 5). The system consists of several key components. At its core, the HEAppE application includes modules for API handling, service orchestration, business logic, data access, file transfer, and HPC connection management. The API module provides REST API endpoints that allow external systems and client applications to interact with HEAppE. The service module coordinates internal operations, while the business logic module implements core functionalities such as job and data management. Secure and efficient interaction with HPC systems is facilitated by the HPC connection module, which manages SSH-based access to remote job schedulers such as PBS and SLURM. To protect sensitive information such as SSH keys and credentials for the access to the HPC, the system incorporates a dedicated secure vault module. Figure 5: HEAppE architecture 3.2.3 Features and Security HEAppE Middleware provides: • Seamless HPC access via a standardized REST API D4.5. HPC API and documentation 18 • Templated jobscripts enabling easy HPC application execution and monitoring without direct access to the cluster • REST API for SLURM or PBS batch job management • Support for other batch schedulers or metaschedulers through plugin interface • Secure credential management for HPC data transfer • Audited remote user-to-HPC access • Secure vault for HPC access credentials • Authentication via Personalized or Robot accounts with Shared or Exclusive access modes • Protection against brute-force attacks, rate limiting, and minimized data exposure • OpenID token role-based authentication HEAppE implements several key security features to ensure secure access to HPC resources. From the authentication point of view HEAppE supports two access modes. First one uses personalized user accounts i.e. one-to-one mapping of user credentials meaning that the HPC compute job executed via HEAppE will be submitted under an actual HPC user account. Second approach utilizes so-called robot accounts where the API users don't have direct access to an HPC infrastructure but HEAppE maps each compute job execution to a selected robot account, thus allowing controlled and restrictive access to the HPC resources. These robot accounts are created for a concrete resource allocation on a specific HPC infrastructure. To prevent arbitrary command execution, users can only run predefined job execution templates called Command Templates. Each template defines a script or executable binary, that will be executed on an HPC infrastructure together with any dependencies or third-party software it might require. 3.2.4 Future work and References HEAppE Middleware is an open-source software with GPL v3 license. It is under constant development with the aim of having quarterly releases containing bug fixes and newly supported features based on feedback from the users and related projects' requirements. HEAppE landing page https://heappe.eu HEAppE source codes https://github.com/It4innovations/HEAppE HEAppE documentation https://heappe.it4i.cz/docs 4. Design of the LUMI AI Factory HPC-API 4.1 Selection of solution components As described in section 3.1.5, FirecREST supports the two first use cases, which are of key importance. In the first use case, the LUMI AI Factory can provide MLOps web services where users access LUMI resources under their own identity. In the second use case, users can configure single-user access on their own laptops and use LUMI as an extension of their local workflows and tools. Both of these can be D4.5. HPC API and documentation 19 supported by a single centralized API service. In principle the third use case could be covered using robot accounts, but initially this usage may not be supported. HeAppE was originally designed to meet the third use case. In that case API users don't have direct access to an HPC infrastructure but HEAppE gives controlled and restrictive access to the HPC resources via robot accounts. Projects can also independently set up HEAppE and tune it for their needs. No centralized installation is being planned for LUMI. 4.2 Implementation plan 4.2.1 Integrating FirecREST with LUMI and CSC infrastructure FirecREST API will be integrated with CSC authentication proxy user-auth. This supports OpenID Connect and the required authorization flows. Another key service is an SSH-CA service. This provides the actual SSH certificate that FirecREST uses to access the supercomputer on behalf of the user. IN 2025, CSC is setting up a SSH-CA service based on the DeiC SSH Certificate Authority Library (https://github.com/wayf-dk/sshca). FirecREST recently received support for this solution. 4.2.2 Kubeflow HPC offloading via FirecREST Kubeflow is a Kubernetes-based MLOps platform that provides a user-friendly way to handle the full life cycle of an ML model, enabling users to develop, orchestrate, and deploy machine learning workflows. As described in Section 1.1, it is a key component of the LUMI AI Factory MLOps environment. LUMI AI Factory Service Center will operate a Kubeflow service on the upcoming LUMI-K container cloud platform. This platform offers a managed, multi-tenant Kubernetes based service running on CPU compute nodes of LUMI. It has no direct access to Slurm or the Lustre file system but will need to access the LUMI HPC partitions via the FirecREST REST API and LUMI-O object store system via the Simple Storage Service (S3) protocol. Users will authenticate using OpenID Connect, enabling seamless single sign-on across the platform. The authentication system will automatically provision access tokens for both FirecREST and LUMI-O services. A key target is to enable Kubeflow to run compute heavy tasks on LUMI-G GPU nodes using this API (HPC offload), while light tasks can remain on LUMI-K. The main use cases that will be offloaded are: • ML workflows: Kubeflow Pipelines are complex ML workflows that can be expressed as a directed acyclic graph (DAG) of tasks. Each task is typically a container that runs in Kubernetes, doing e.g. data pre-processing, training, evaluation, etc. The primary goal is to enable these tasks to be offloaded. • Hyperparameter tuning: The Katib component can be used for hyperparameter tuning by launching many training jobs in parallel. A potential future goal is to offload these to HPC. Some use cases for Kubeflow will not be offloaded to HPC: • Interactive Jupyter notebooks: Typically used for data analysis and model development tasks. The LUMI supercomputer already provides a web interface where Jupyter notebooks can be run on, hence we will in this work not implement HPC offloading for this component. Some data processing and model development can still be done also on the LUMI-K nodes. D4.5. HPC API and documentation 20 • Serving models: The KServe component is an inference service component for serving the developed models. In the LUMI AI Factory MLOps environment the Aitta service is the main model serving platform; hence no integration of KServe is currently planned. 4.2.2.1 Implementation plan The goal is to keep the pipeline definition and user experience as close to normal as possible. The most straightforward approach is to implement a custom Kubernetes pipeline component. This allows the user to define the container image and commands to be run, a set of input data (e.g. S3 path), a S3 path where output data is to be placed, and finally a specification of the HPC resources required for the computation (number of GPUs, max job time, accounting project, etc.). When the component is launched as part of a Kubeflow pipeline, it runs as a lightweight pod in LUMI-K. This pod does not perform the actual computation but instead launches and monitors a job on the HPC system to do so. It also transfers data to and from the HPC system. All these tasks can be performed using FirecREST REST API calls. In addition to a generic component, it is also possible to develop more specialized components for specific tasks such as training, making the components easier to use. An important consideration is that the LUMI HPC partitions are only able to run Apptainer or Singularity containers. This is not a major problem since they can transparently convert Docker containers into their specific formats when running. A key task is still to provide a good set of containers that can be used for parallel large-scale tasks on the HPC system, that are able to efficiently utilize GPUs across multiple nodes and MPI. The work to develop this set of containers is described in Deliverable 4.4. in detail. 4.2.2.2 Alternative approaches and future work It should be possible to use a similar approach as the Kubeflow pipelines in Katib, Kubeflows hyperparameter tuning framework, where FirecREST is used to offload trial jobs to HPC. An interesting alternative to the pipeline component approach mentioned above is to implement a generic custom resource (CRD) representing FirecREST. Such a FirecRESTJob CRD would function like existing job types (TFJob, PyTorchJob) in Kubeflow. Technically, this also requires implementation of a controller that watches FirecRESTJob resources, translates specifications to FirecREST API calls, manages job lifecycle, and reports status information back to the Kubernetes API. The benefits of the CRD implementation are that it enables us to implement the FirecREST integration once instead of duplicating that logic across several pipeline components. Pipeline components then become thin wrappers that create FirecRESTJob resources and wait for completion. Another main benefit is that they would integrate much better into Kubernetes, and normal tooling can be used for monitoring and management. However, as this is likely a more complex task than adding a generic pipeline component, the latter option will be explored first, and investigate implementation of a CRD in later stages of the project.