D2.2 : Data and Workflow Management Toolbox Alpha Status Report

D2.2 Data and Workflow Management Toolbox Alpha Status Report BADW-LRZ Public ©EXA4MIND 2023–2025 D2.2 – Data and Workflow Management Toolbox Alpha Status Report Deliverable Properties Version v1.0 Dissemination level PU Deliverable type R Work package WP2 Task T2.1 Due date 31-12-2023 Submission date 22-12-2023 Deliverable lead BADW-LRZ Authors Name Mohamad Hayek (BADW-LRZ), Martin Golasowski (IT4I@VSB), Stephan Hachinger (BADW-LRZ), Jan Martinovič (IT4I@VSB), David Číž (IT4I@VSB), David Hurych (VALEO), Piyush Harsh (TERRAVIEW) Reviewers Tomáš Martinovič (IT4I@VSB), and Jan Zahradník (VALEO) Abstract The EXA4MIND project connects pre-eminent databases and data management systems to supercomputing systems and European Data Spaces as well as the world of FAIR research data. The core purpose of this endeavour is running next-generation Extreme Data workflows, with emphasis on data analytics, Machine Learning / Artificial Intelligence, or classical simulations. This deliverable reports on the Data and Workflow Management Toolbox provided for this purpose, building upon the successful LEXIS Platform (delivered by the H2020 project, GA 825532). Furthermore, it illustrates the first workflows run by our application cases at supercomputing centres as a basis for the milestone MS5 First Data-driven Workflows have been Executed using Systems at Supercomputing Centres. Keywords Extreme Data; Workflow Control; Data Management; EXA4MIND Toolbox 222 D2.2 – Data and Workflow Management Toolbox Alpha Status Report Document Revision History Version Date Description of Change Contributor(s) v0.1 17-10-2023 Deliverable outline Mohamad Hayek (BADW-LRZ) v0.2 21-11-2023 Added first parts Stephan Hachinger (BADW-LRZ) v0.3 05-12-2023 First version of all main text parts completed Stephan Hachinger, Mohamad Hayek (BADW-LRZ); David Číž (IT4I@VSB), David Hurych (VALEO), Piyush Harsh (TERRAVIEW) v0.4 06-12-2023 Added conclusions Stephan Hachinger (BADW-LRZ) v0.5 13-12-2023 Extended introduction text and Sections 2 and 3, added details of WP5 workflows Martin Golasowski, Jan Martinovič (IT4I@VSB) v0.6 21-12-2023 Consolidated version after the review, WP6 section updated Mohamad Hayek, Stephan Hachinger (BADW-LRZ), Martin Golasowski (IT4I@VSB) v1.0 22-12-2023 Final check of the deliverable Kateřina Slaninová, Jan Martinovič (IT4I@VSB) Disclaimer Information, documentation and Figures available in this deliverable are provided by the EXA4MIND project’s consortium funded by a European Union’s Horizon Europe Research and Innovation programme under grant agreement No 101092944. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the granting authority can be held responsible for them. Copyright Notice ©EXA4MIND 2023–2025 Dissemination Levels PU Public Public, fully open, e.g. website SEN Sensitive Confidential to EXA4MIND project and Commission Services 322 D2.2 – Data and Workflow Management Toolbox Alpha Status Report Deliverable Types RDocument, report (excluding periodic and final reports) DEM Demonstrator, pilot, prototype, plan designs DEC Websites, patent filings, press and media actions, videos, etc. OTHER Software, technical diagrams, etc. 422 D2.2 – Data and Workflow Management Toolbox Alpha Status Report Table of Contents Executive Summary ....................................... 9 1 Introduction .......................................... 10 2 Current State of the EXA4MIND Toolbox ........................ 11 2.1 WorkflowOrchestration ............................... 11 2.2 Distributed Data Management . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.3 User Interfaces and Resource Management . . . . . . . . . . . . . . . . . . . 14 3 Development of EXA4MIND EDD and AQIS Features – Integration with the Toolbox ............................................. 15 3.1 EDD: Relation to Data and Workflow Management Toolbox . . . . . . . . . 15 3.2 EDD Interfaces for Usage, FAIRification, and HPC-based Processing of Data 16 4 First Data-driven Workflows Executed at Supercomputing Centres ....... 16 4.1 Scientific Application Case (WP4) Examples . . . . . . . . . . . . . . . . . . . 17 4.1.1 User-defined Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 4.1.2 Comparison with Dxperimental Cata . . . . . . . . . . . . . . . . . . . 17 4.2 Industry Application Case (WP5) Examples . . . . . . . . . . . . . . . . . . . 18 4.3 SME Application Cases (WP6) Examples . . . . . . . . . . . . . . . . . . . . . 18 4.3.1 Agriculture ................................... 18 4.3.2 Healthcare ................................... 20 5 Conclusion .......................................... 20 6 References .......................................... 21 522 D2.2 – Data and Workflow Management Toolbox Alpha Status Report List of Figures Figure 1 Existing LEXIS Platform components and extension through EXA4MIND. .. 10 Figure 2 Airflow UI with LEXIS Provider. .......................... 12 Figure 3 LEXIS Workflow as Airflow DAG. ......................... 13 Figure 4 Example of Processed Driving scene via WP5 Workflow with Depicted Results of Eetected and Segmented Object Instances (Brightness Enhanced Inside of Polygon) by AI Model. .......................... 19 List of Tables No tables are given. 622 D2.2 – Data and Workflow Management Toolbox Alpha Status Report Abbreviations AI Artificial Intelligence AAI Authentication and Authorization Infrastructure API Application Programming Interface AQIS Advanced Querying and Indexing System CPU Central Processing Unit (processor) CRUD Create Read Update Delete operations on a data DAG Directed Acyclic Graph DBMS Database Management System DDI Distributed Data Infrastructure DoA Description of Action EDD Extreme Data Database ETL A pipeline with extract, transform and load tasks GA Grant Agreement GPU Graphics Processing Unit (accelerator) HPC High Performance Computing HTTPS Hypertext Transfer Protocol with SSL IaaS Infrastructure as a Service iRODS Integrate Rule-Oriented Data System JWT JSON Web Token JSON JavaScript Object Notation LEXIS Large Scale Executions for Industry and Society - H2020 project OIDC OpenID Connect POSIX Portable Operating System Interface REST Representational State Transfer S3 Simple Storage Service - object storage / REST API based on AWS standard SMC Soil Moisture Content SSL Secure Sockets Layer SQL Structured Query Language WP Work Package YAML Yet Another Markup Language 722 D2.2 – Data and Workflow Management Toolbox Alpha Status Report Table of Partners Short Name Partner IT4I@VSB IT4Innovations at VSB – Technical University of Ostrava (IT4Innovations) https://www.it4i.cz/en BADW-LRZ Leibniz Supercomputing Centre (LRZ) of the Bavarian Academy of Sciences and Humanities https://www.lrz.de/english/ METU Middle East Technical University (METU) https://www.metu.edu.tr/ AUSTRALO AUSTRALO Interinnov Marketing Lab https://www.australo.org/ EURAXENT Euraxent https://www.linkedin.com/company/euraxent/ CVUT Czech Institute of Informatics, Robotics and Cybernetics at the Czech Technical University (CIIRC CTU) https://www.ciirc.cvut.cz/ VALEO Valeo https://www.valeo.com/en/czech-republic/ TERRAVIEW Terraview https://www.terraview.co/ ALTRNATIV Altrnativ https://altrnativ.com/?lang=en VALEO.AI Valeo.ai https://www.valeo.com/en/valeo-ai/ 822 D2.2 – Data and Workflow Management Toolbox Alpha Status Report Executive Summary To facilitate next-generation Extreme Data workflows, the EXA4MIND Platform needs proper workflow and data flow management. Such management enables computational workflows with analytics, Artificial Intelligence or simulation core tasks across different specialised systems at supercomputing centres – from GPU-equipped clusters, over classical CPU machines to versatile “cloud-native” infrastructure. In particular, it enables application-case partners who are not used to supercomputing systems to deploy their data-intensive workflows in a user-friendly, but well-controlled and systematic manner. To this end, Task 2.1 has been foreseen to consolidate the experience and previous open-source systems available to the project partners and to provide workflow management for EXA4MIND – which, in the end, we have been able to base on the LEXIS Platform (developed in the H2020 project LEXIS, GA 825532). This deliverable reports the results of Task 2.1 up to M12 – the ”Data and Workflow Management Toolbox Alpha Status”. It first introduces our basic ideas for the toolbox, including some LEXIS Platform background (Section 1), and then elaborates (Section 2) on the toolbox in terms of workflows, data management and user interfaces/resource management. The envisaged relation of the toolbox to the EXA4MIND Extreme Data Database, and the co-evolution with this toolbox is sketched in Section 3. Finally, we expand somewhat (Section 4) on the first workflow examples from EXA4MIND run at our supercomputing centres (IT4I@VSB, BADW-LRZ), which is the basis for certifying milestone MS5 (M12) and conclude (Section 5). 922 D2.2 – Data and Workflow Management Toolbox Alpha Status Report When this source or target, together with a formalised AQIS query, is given instead of a conventional file system with a directory path, EDD will address AQIS to receive or send the data through the specified query. This is one of the central ideas in EXA4MIND which shall make it possible to facilitate data connectivity between computing clusters, EDD and European Data Spaces. 3.2 EDD Interfaces for Usage, FAIRification, and HPCbased Processing of Data EXA4MIND features a rich set of preprocessors and adaptors as well as use-casecentric APIs (e.g. for web portals to attach to) to support the application cases in data usage. Extending the interfacing, data exchange mechanisms and FAIRification mechanisms of EDD to interact with European Data Ecosystems will go far beyond what was possible with the LEXIS DDI. In particular, the publication of selected data contained in DBMS will be facilitated, which constitutes a semantic challenge not foreseen in the LEXIS Platform. Another important part of the EXA4MIND concept and architecture development is integration models for using EDD data in HPC environments. To this end, the usage of DBMS or DBMS data on HPC systems is being investigated in Task 2.3. In principle, HPC codes can interact with DBMS data in three ways: (i) DBMS data are dumped/reingested to be used in the form of normal files in the HPC context, (ii) DBMS data are dumped/re-ingested, but on the HPC system they are used via a local DBMS server, started on each computing node, at least for read access, or (iii) a central DMBS server is started on a reserved node in the HPC-system context. These mechanisms are currently being evaluated, and on the BADW-LRZ Linux Cluster HPC system, mechanisms (i) and (ii) seem at least feasible. Further tests have to concentrate on feasibility and performance, to recommend one or more methods for HPC access to DBMS data to WP4, WP5 and WP6. 4 First Data-driven Workflows Executed at Supercomputing Centres To summarise the concepts from this deliverable, we describe initial workflows from our application cases (WP4-WP6) that will make use of the data and workflow management toolbox. The first test runs of these workflows, e.g. with manual control have already been executed at the supercomputing centres. Clearly – having presented the alpha version of the toolbox here – the task of porting these workflows to use the toolbox will be the next challenge for EXA4MIND. We are confident that tackling this challenge will bring significant benefits in terms of workflow efficiency. 16 22 D2.2 – Data and Workflow Management Toolbox Alpha Status Report 4.1 Scientific Application Case (WP4) Examples WP4 has become very well acquainted with the supercomputing infrastructure operated by IT4I@VSB, mainly the Karolina cluster. Below, the two deployed and executed workflows are described. 4.1.1 User-defined Analysis The first workflow models the most basic use case which is user-defined analysis. The workflow performs a series of analyses on a given topology and trajectory file using the Ptraj program (Roe and Cheatham III 2013). The workflow consists of the following steps: ◆The workflow is initiated by running the run-ptraj_1.sh shell script. Currently, as no database exists, there are hardcoded paths to the trajectory and topology files, stored in the HPC project directory. ◆The shell script executes the ptraj program with the ptraj.in trajectory command file, which contains the commands for the desired analyses. ◆The ptraj program outputs several .dat files, which contain the results of the analyses, such as the pucker or chi angle. The workflow is specific to the input files and does not have any parameters that the user can adjust. The workflow requires the ptraj program to be installed and accessible on the HPC system. 4.1.2 Comparison with Dxperimental Cata The second workflow performs a comparison of analysis of results with experimental data. As such, this workflow can be chained with the previous one and we do reuse the results of the previous workflow for the comparison. The second workflow consists of the following steps: ◆The workflow is initiated by running the md-exp-comp.sh shell script, which takes no input arguments. ◆The shell script performs several operations on the data files generated by the first workflow, such as merging, filtering, calculating, and comparing. ◆The shell script outputs several .dat files, which contain the comparison results between the experimental and simulated data, such as NOE, uNOE, J-coupling, and RMSD. The workflow requires the data files from the first workflow to be present in the same directory. 17 22 D2.2 – Data and Workflow Management Toolbox Alpha Status Report 4.2 Industry Application Case (WP5) Examples The WP5 workflows are centred around AI workflows using HPC. They consist of both inference and training tasks and are part of a high-level pipeline that should produce ML models trained on private data from VALEO with full traceability. As WP5 deployed its applications on IT4@VSB HPC cluster Karolina, we tested two options for data transfers. The first one was using the iRODS client on the VALEO side to transfer data to the iRODS zone at IT4I@VSB. As iRODS uses an array of high TCP ports for parallel data transfer, it interferes with network security policies and measures used by VALEO and turned out not feasible. The second option was chunked S3 upload, which is based on HTTPS, therefore using only a single TCP port; at the same time, VALEO already uses it internally for data transfers. Therefore, we used the S3 protocol to transfer the first batch of VALEO data to IT4I@VSB. This corresponds directly with the planned extension of EDD with the S3 support. The initial workflow executed by VALEO is part of the pre-processing anonymisation process, which performs image segmentation and blurring of sensitive parts like human faces or licence plates. The workflow uses AI models to extract various scene attributes. We successfully developed the first (more coming) AI model that combines object detector and instance segmentation tasks. This AI model creates bounding boxes around objects of 10 different classes (traffic light, traffic sign, person, rider, car, truck, bus, train, motorcycle, bicycle) and marks their presence in images as presented in Figure 4. The EXA4MIND EDD and AQIS can help in this regard to streamline the image data transfers and especially storing the bounding boxes and instance segmentation data derived by the preprocessing step in specialised databases. The results of the workflow with the BLIP AI model (Li et al. 2022) used for feature extraction executed on Karolina are as follows. The workflow is roughly divided into three stages – the ”BLIP feature extraction”, ”image loading”, and the ”JSON file saving”. Performed on a single A100 GPU with 40GB of memory. The input data consumed less than 20GB of the GPU memory (images loaded one by one), and most of the memory was consumed by the BLIP model. During testing, each image was loaded just once and several BLIP feature vectors were extracted. One feature vector for the full image and one feature vector for each Bounding Box (BBox) encapsulating a particular object in the image. The bounding boxes were extracted in advance by the Yolo v8 detection model (Ultralytics 2023). Yolo v8 detection was not part of these measurements, we just used the BBoxes for input. The input data set contains 3,583 images (2MPix each). 4.3 SME Application Cases (WP6) Examples 4.3.1 Agriculture TERRAVIEW provides two workflows within its Aquaview framework: The Soil Moisture Content (SMC) computation workflow and the moisture modelling workflow. Components of these workflows are currently being shifted to be run at the supercomputing 18 22 D2.2 – Data and Workflow Management Toolbox Alpha Status Report Figure 4: Example of Processed Driving scene via WP5 Workflow with Depicted Results of Eetected and Segmented Object Instances (Brightness Enhanced Inside of Polygon) by AI Model. centres. While usage of LRZ infrastructure – including the Terrabyte satellite-data mirror infrastructure1– is also envisaged, first runs will be executed at IT4I@VSB. TERRAVIEW will use HPC capabilities to accelerate the SMC computation workload. To achieve this, TERRAVIEW will deploy the data ingestion workflow to download the satellite scenes from USGS (U.S. Geological Survey 2023) periodically, to create a cache at IT4I@VSB. Thus, the SMC computation workflow will use this cache to obtain the satellite imagery, rather than locally perform the download using TERRAVIEW’s computational resources. The longer-term aim is here to offload computationally costly and storage-heavy parts of the SMC workflow to supercomputing infrastructure. For the moisture modelling workflow, Aquaview employs two modelling workflows. The ground models correlate historical weather sequences with satellite Soil Moisture Content maps (SMCMs) and create a virtual moisture probe (model) for each reading pixel. The depth models correlate historical weather sequences with moisture data from probes in vineyards and create a virtual depth moisture probe (model) per the actual physical probe. Both modelling pipelines are data parallel and can run independently. Also, all model training processes within these pipelines are data parallel and can be run in parallel. Historical weather data is stored in files representing virtual weather stations. Historical Soil Moisture Content Maps (SMCMs) are generated 1Terrabyte satellite-data mirror infrastructure: https://stac.terrabyte.lrz.de,https://www. dlr.de/eoc/en/desktopdefault.aspx/tabid-11882/20871_read-84140/ 19 22 D2.2 – Data and Workflow Management Toolbox Alpha Status Report through Aquaview’s site onboarding workflow (DAG). 4.3.2 Healthcare ALTRNATIV is currently, with the help of WP2, deploying its ETL worker on the LRZ IaaS-Cloud infrastructure. This software piece takes charge of the dataflow/workflow preparation and executes different extract-transform-load tasks (pipeline) on datasets. The tasks executed can include data mining or even ML/DL-based inference. Furthermore, ALTRNATIV has prepared a connector to facilitate data management (and later workflow management) in the context of their ETL pipeline with the LEXIS Platform and the Data and Workflow Management Toolbox. 5 Conclusion In this deliverable, we have laid out how we leverage our experience with the LEXIS Platform to construct the Data and Workflow Management Toolbox for the EXA4MIND Extreme Data workflows. We have discussed the work performed on the toolbox so far, to provide a baseline for its integration with EXA4MIND EDD and AQIS. In particular, we have described the extensions motivated by an analysis of the EXA4MIND application cases. The concept will be gradually refined and implemented in the next phase, especially as concrete DBMS will be selected and integrated within EDD for the application cases. To illustrate how the toolbox will be used in the future, the third part of the deliverable has discussed the first workflows run by the application cases on the supercomputing infrastructure. These workflows are also vital for the acceptance of the project milestone MS5. In the project phases to come, EXA4MIND aims to gradually integrate each application case with EDD and AQIS as their features become available. 20 22 D2.2 – Data and Workflow Management Toolbox Alpha Status Report 6 References Amstutz, Peter et al. (2016). Common Workflow Language, v1.0. DOI: 10.6084/m9. figshare . 3115156 . v2. URL: https : / / doi . org / 10 . 6084 / m9 . figshare . 3115156.v2. Brogi, Antonio, Jacopo Soldani, and PengWei Wang (2014). “TOSCA in a Nutshell: Promises and Perspectives.” In: Service-Oriented and Cloud Computing. Ed. by Massimo Villari, Wolf Zimmermann, and Kung-Kiu Lau. Berlin, Heidelberg: Springer, pp. 171–186. ISBN: 978-3-662-44879-3. DOI: 10.1007/978-3-662-44879-3_13. Elastic (2023). Elasticsearch.https://www.elastic.co/elasticsearch/, Cited 01 Dec 2023. EUDAT CDI (2023a). B2ACCESS-EUDAT.https://www.eudat.eu/services/b2access, Cited 30 Nov 2023. EUDAT CDI (2023b). B2HANDLE-EUDAT.https://www.eudat.eu/services/b2handle, Cited 30 Nov 2023. EUDAT CDI (2023c). B2SAFE-EUDAT.https://www.eudat.eu/services/b2safe, Cited 30 Nov 2023. EUDAT CDI (2023d). EUDAT.https://www.eudat.eu, Cited 30 Nov 2023. EXA4MIND (2023a). Deliverable D1.1 Application cases and Architecture Requirements. EXA4MIND (2023b). Deliverable D2.1 Extreme Data Flow Patterns – Report. GÉANT Project and Association (2022). MyAccessID Home – MyAccessID – GÉANT federated confluence.https://wiki.geant.org/display/MyAccessID/MyAccessID+ Home, Cited 30 Nov 2023. Golasowski, Martin and et al. (2022). “The LEXIS Platform for Distributed Workflow Execution and Data Management.” In: HPC, Big Data, and AI Convergence Towards Exascale. Boca Raton (FL): CRC Press, pp. 17–35. ISBN: 978-1-003-17666-4. DOI: 10.1201/9781003176664-2. Hachinger, Stephan et al. (2022). “Leveraging High-Performance Computing and Cloud Computing with Unified Big-Data Workflows: The LEXIS Project.” In: Technologies and Applications for Big Data Value. Ed. by Edward Curry et al. Cham: Springer International Publishing, pp. 159–180. ISBN: 978-3-030-78307-5. DOI: 10 . 1007 / 978-3-030-78307-5_8. URL: https://doi.org/10.1007/978-3-030-783075_8. Lecarpentier, Damien et al. (2013). “EUDAT: A New Cross-Disciplinary Data Infrastructure for Science.” In: Int. J. Digit. Curation 8.1, pp. 279–287. LEXIS (2023a). LEXIS Platform documentation.https://docs.lexis.tech/, Cited 01 Dec 2023. LEXIS (2023b). Py4lexis - Python Client for Lexis Platform.https://opencode.it4i.eu/ lexis-platform/clients/py4lexis, Cited 01 Dec 2023. Li, Junnan et al. (2022). “BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation.” In: CoRR abs/2201.12086. arXiv: 2201.12086. URL: https://arxiv.org/abs/2201.12086. Munke, Johannes et al. (2022). “Data System and Data Management in a Federation of HPC/Cloud Centers.” In: HPC, Big Data, and AI Convergence Towards Exascale. Boca Raton (FL): CRC Press, pp. 59–77. ISBN: 978-1-003-17666-4. DOI: 10.1201/ 9781003176664-4. 21 22 D2.2 – Data and Workflow Management Toolbox Alpha Status Report OASIS (2023). OASIS TOSCA.https://www.oasis-open.org/committees/tosca/, Cited 01 Dec 2023. Red Hat (2023). Keycloak.https://www.keycloak.org/. Redis Labs (2023). Redis.https://redis.io/, Cited 01 Dec 2023. Roe, Daniel R and Thomas E Cheatham III (2013). “PTRAJ and CPPTRAJ: software for processing and analysis of molecular dynamics trajectory data.” In: Journal of chemical theory and computation 9.7, pp. 3084–3095. Scionti, Alberto et al. (2020). “HPC, Cloud and Big-Data Convergent Architectures: The LEXIS Approach.” In: Complex, Intelligent, and Software Intensive Systems. Ed. by Leonard Barolli, Farookh Khadeer Hussain, and Makoto Ikeda. Cham: Springer International Publishing, pp. 200–212. ISBN: 978-3-030-22354-0. DOI: 10 . 1007/ 978-3-030-22354-0\_19. The Apache Software Foundation (2023). Apache Airflow.https://airflow.apache.org/, Cited 01 Dec 2023. U.S. Geological Survey (2023). USGS.gov | Science for a changing world.https://www. usgs.gov/, Cited 24 Nov 2023. Wilkinson, Mark D. et al. (2016). “The FAIR Guiding Principles for scientific data management and stewardship.” In: Scientific data 3. Xu, Hao et al. (2017). iRODS primer 2: Integrated Rule-Oriented Data System. Williston, VT: Morgan & Claypool Publishers. DOI: 10.2200/S00760ED1V01Y201702ICR057. Ultralytics (2023). Ultralytics YOLOv8.https://yolov8.com, Cited 9 Nov 2023. 22 22 This is the version of the deliverable before the review by European Commission.