An overview on DAPHNE4NFDI activities at DESY

An overview on DAPHNE4NFDI activities at DESY Status report and who is who DAPHNE4NFDI Annual meeting 2024 –March 13th 2024 Regina Hinzmannawith contributions of Lisa Amelungb, Noel Bartha, Anton Bartyb, Igor Khokhriakovb, Linus Pithanc, Christoph Rosemannb, Parthasarathy Tirumalaib aDESY IT-InFa, bDESY FS-SC, cDESY FS-EC Page 2| An overview on DAPHNE4NFDI activities at DESY | Regina Hinzmann Who is who at DESY –involvement in DAPHNE4NFDI Many groups –one goal www.uantwerpen.be/scream •Photon Science (FS – interim head: Franz Kärtner) • Scientific Computing (SC – lead: Anton Barty) • Lisa Amelung, Igor Khokhriakov, Partha(sarathy) Tirumalai Nallam Chakravaty, Christoph Rosemann • Experiment Control (EC – lead: Linus Pithan, formerly: Thorsten Kracht) • Jan Kotanski, Anjali Aggarwal • FLASH contact: Stefan Düsterer (FLASH Photon Diagnostics and Controls) • PETRA contact: Florian Bertram (P08) •High Energy Physics (FH – head: Beate Heinemann) • Information Technology (IT – head (tba): Philipp Neumann) • Information Fabrics (InFa – lead: Peter van der Reest) • Regina Hinzmann, Noel Barth, Johannes Reppin • Research and Innovation in Computing (RIC – lead: Patrick Fuhrmann) • Tim Wetzel, Paul Millar, Uwe Jandt • Library (L – head, Martin Köhler) legend:'meet'us'–'we'are'around! Page 3 One goal or… … why and how SciCat came to DESY | An overview on DAPHNE4NFDI activities at DESY | Regina Hinzmann •We need to find back data produced at DESY • We aim for FAIR data to cover DESY data for its entire life cycle: from the moment of data-taken until long term archive • DESY needs a catalogue. It was decided in Jan 2022 to go for SciCat. Some reasons: active and broad community behind, already in use by many labs (MAX IV, PSI, ESS, RFI...). Benefit from their experience. • Since then: •Setup an OpenStack and Kubernetes infrastructure for better maintainability of several SciCat instances • Established test instances at PETRAand FLASH beamlines (demonstrator beamlines) •Contribute to code development of SciCat • Established since Sep 2023 regular meetings for reports and communication (SciCat Technical and SciCat General) •Goal: Offer a reliable and stable service for the PETRAand FLASH beamline experimentalist for internal use, contribute to FAIR data and to open data access for external use. Public'indico meeting'notes'etc. Regina'Hinzmann'(IT-InFa) Page 4 SciCat instances at DESY (examples) Set up based on OpenStack and Kubernetes infrastructure by IT-InFa •https://scicat-p08-test2.desy.de •https://scicat-flash-test2.desy.de •https://scicat-rockit-p65-test1.desy.de •https://scicat-p05.desy.de | An overview on DAPHNE4NFDI activities at DESY | Regina Hinzmann •https://scicat-fsec-dev.desy.de •https://scicat-desyfs-test1.desy.de SciCat at beamlines (demonstrator instances) FS instances for DOIs, alternative metadata ingestion (development instances) •https://public-data.desy.de For IT-RIC group, see Tim Wetzels talk Page 5 Work areas related to SciCat Highlight current major work areas •Data model for Metadata standardization across DESY photon science beamlines: On basis of TA1 & TA2 results, a machineactionable description of the scientific metadata in SciCat, e.g. for validation, is developed (FS: Anjali A, Linus P, Fabiano L). •DOI minting: idea is to provide DOIs using SciCat (N.N., IT) • FS and L agreed on condition for issuing a DOI: 1 Beamtime = 1 dataset, ultimately: User selects datasets within SciCat and initiates the minting process (“user awareness”). • But need a functionality to make a list of datasets, not yet migrated to the new backend (“jobs” = cart functionality: communication of SciCat to an external service) •DOOR (DESY’s proposal management system, DESY Online Office for Research with Photons) accounts need to be integrated into federated DESY accounts as used in SciCat; stepwise procedure (FS) •Contribution to SciCat code development •“Jobs”: contributed to schema definition and writing tests (Regina H, led by Daphne v Dijk) • Frontend: User experience improvement (lead release Search UI by Igor K) | An overview on DAPHNE4NFDI activities at DESY | Regina Hinzmann Page 6 Frontend community issues Goal: new UI search release | An overview on DAPHNE4NFDI activities at DESY | Regina Hinzmann Igor'Khokhriakov'(FS-SC) More DESY contributions Page 8 • Pyinstaller executables using Many Linux containers • Plug2: A simple plotter for XPCS @P10 • RSAP: Reciprocal Space Maps Program • Nuclear Elastic X-ray scattering Universal Software (Nexus) • Windows 11 and MacOS (14.3.1Sonoma) builds • Many linux dynamically linked Python 3.6 – 3.12 container image Continuous Delivery & Deployment at DESY FS-SC Applications under Gitlab CI/CD (continuous integration/ continuous deployment) | An overview on DAPHNE4NFDI activities at DESY | Regina Hinzmann Software packaging tools • CrystFEL1 • Continuous deployment on Maxwell • Updates to Singularity/Apptainer container image • Updates to Homebrew recipe for MacOS To Do • Setup for CI/CD for other software projects • Automated installation testing of packages and run sanity tests • Identify & explore more software packaging tools & techniques Parthasarathy'Tirumalai'(FS-SC) Page 9 Summary and Outlook Still some way to go, but we’re underway! •After two years of SciCat at DESY we have: • A cluster infrastructure based on Kubernetes and OpenStack, documentation too is foreseen (April ‘24) • Authenticate to SciCat via OIDC (AAI, keycloak -SSO) • More than 10 SciCat instances for different internal purposes available, mostly to exercise workflows (e.g. DOI) • We constantly gain experience through collaboration with other SciCat developers •IT-InFA: Noel, Johannes, Peter, Regina • Encouraging work has just emerged on metadata model development in a machine readable format standardized across all DESY beamlines (FS: Linus, Anjali & Fabiano) • Other contributions • Possibly curated analysis software repository as part of TA3 (Partha) Outlook/conclusion • Continue to tackle concrete problems and document progress within the SCT and SCG frames and encourage communication. | An overview on DAPHNE4NFDI activities at DESY | Regina Hinzmann This'is'merely'a'selection'of'a'few'activities'and'people'involved.'