Improving HPC usage in ESS by FAIR data & compute services

NFDI4Biodiversity and NFDI4Earth are funded by the German Research Foundation (DFG). NFDI4Biodiversity DFG project number 442032008. NFDI4Earth DFG project number 460036893 www.nfdi4earth.de |www.nfdi4biodiversity.org Mission of our group Improving HPC usage in ESS by FAIR data & compute services 1DLR –Oberpfaffenhofen, 2AWI –Bremerhaven,3BADW-LRZ –Garching b.M., 4Technische UniversitätDresden, 5DKRZ –Hamburg Eberle, J.1, Frickenhaus, S.2, Fritzsch, B.2, Hachinger, S.3, Humbert, A.4, Koldunov, N.2, Löwer, N.4, Müller-Pfefferkorn, R.4, Munke, J.3, Thiemann, H.5–IG ”High-Performance Computing in Earth System Sciences” in NFDI4Earth Our concept paper of 2022 stated our mission and analysed two typical HPC ESS use cases to identify challenges in HPC usage and in FAIR handling of the results. The IG focuses on making FAIR possible in HPC, and also to enable Earth System scientists to use HPC. The paper gives 17 recommendations on how to follow these goals. FAIR HPC in ESS is challenging! What are important issues for you? What are your solutions? Tell us with a post-it! Reality check: CAPICE Pilot –Ice sheet models with Distributed HPC & FAIR data How do you see that AI can help to make data FAIR? And in turn, what do you consider “Prio1” steps to “FAIRify” AI data? Tell us with a post-it! Improve reproducibility and foster FAIR principles in HighPerformance Computing (HPC) for Earth System Sciences •Metadata/interoperability/reproducibility concepts •Findability & accessibility of huge data sets; federated access 1. An infrastructure registry for NFDI4Earth to find matching computing resources 2. Approaches to access computing resources through Single Sign On 3. S eamless cross-centre and crosssystem computing access for users including accounting 4. I nteroperability & interconnection of computing systems 5. Robust computing environments for re -usability and reproducibility (e.g. containerisation ) 6. Establishing (meta - )data formats in collaboration with standardisation bodies, RDA, etc. 6.a) Max. ~10 general storage formats 6.b) Max. ~10 metadata formats 6.c) NFDI4Earth recommends these formats 6.d) Support for these formats at all computing centres in NFDI4Earth 7. Standardised software environments at computing centres, documented within NFDI or NFDI4Earth 8. Recommendations on RSE and software packaging 9. Recommendations on quality control 10. Political work towards viable solutions across state/national borders 11. Issue NFDI4Earth recommendations on efficient data access and transfer for Big Data 12. Removal of administrative barriers regarding federated data/compute access 13. Data-portal and storage solutions with spatiotemporal and layer -/field-based data selection & download 14. Establish standardised (meta-)data interface to instituteor subdiscipline-specific interfaces 14.a) Metadata interfaces, starting with OAI - PMH 14.b) Advance usage of standard data interfaces, e.g. S3 15. Clarification and NFDI4Earth index for LTA 16. Interactive HPC including checkpointing 17. Concepts for services reacting on data events (e.g. alerting) •Ice sheet modelling + EO data + Calving front prediction •Federate storage/compute across DKRZ/LRZ: challenging! •Principles proven viable and model example made FAIR infrastructures involved Zenodo/DOI Links: Artificial Intelligence (AI) models have seen an incredible rise as complement to classical simulations or as tools for data processing and interpretation. HPC centres can enable large-scale generative or discriminative AI and Machine Learning (ML) applications, including Large Language Models (LLMs). In the light of such developments, we issued an update to our concept paper, status-checking and extending our 2022 recommendation catalogue in these four fields: AI models are data-driven and data-hungry. Flexible data lakes for sharing data and accessing them via unified protocols (also cross-site) are thus essential. We recommend optimizing data management workflows and quality assurance and settling on well-acceptedaccessprotocols(e.g.S3). Data indexes and catalogues (such as STAC) enable precise data selection before loading. Beyond their role in FAIR data, they help to minimizetransfersandmaximizeefficiency. Making AI models FAIR requires preservation and publication of training dataand trained models. An extreme challenge with the fast development cycles and the high volumes of ESS data. ESS FAIR efforts need a lot of storage. However, HPC centres tend to only offer schemes for awarding corehours based on scientific excellence. We propose to grant storage for excellent FAIR data projects –independently of computing plans. This will facilitate data reuse at scale. Our first concept paper –2022 Reality check: CAPICE pilot in NFDI4Earth –ice sheet models with distributed HPC & FAIR Challenges in FAIR HPC in ESS / FAIR helps AI & vice versa –how? Give us your views ! Have a look at our concept paper (original & update): doi:10.5281/zenodo.6565405 ; doi:10.5281/zenodo.14796888 Contacts / IG Leads: [email protected] [email protected] AI&LLMs: a changing world vs. HPC Challenge I) Data Lakes for Data Science Challenge II) Indexes & Catalogues Challenge III) FAIR AI in ESS Challenge IV) Data/Storage Projects at HPC Centres