Supercomputer "Bura" as a software processing centre for Rubin-related science
Abstract
Large long-term surveys such as LSST at Vera C. Rubin Observatory will obtain a huge, petabyte-scale of data that would need substantial computer resources in order to be processed, analysed and visualised. Therefore, supercomputers and high- performing computing (HPC) facilities play an important role in Rubin/LSST. With the start of Rubin/LSST observations, and the current availability of terabyte-size DP1, as well as of DP2 in the next year, real observations can be used for scientific analysis and final preparation for the huge amount of data expected in Data Release 1. As a part of LSST Vera Rubin Observatory in-kind contribution of Croatian participation group (CPG), HPC supercomputer ’Bura’ at University of Rijeka acts as as a software processing centre. In this talk, the main features of HPC ’Bura’ will be presented, as well as the recent updates, training opportunities and possibilities of its use by Rubin/LSST scientific community, mainly (but not exclusively) in the areas covered by Transients and Variable Stars (TVS), Solar System and Stars, Milky Way & Local Volume (SMWLV) scientific collaborations. Various possible scientific cases, running or planned, such as periodicity determination in variable sources, Solar systes survey simulations, hydrodynamic pulsation models of LPVs, determination of stellar distances by photo-D, where ’Bura’ could be or has been used, and interesting for wider scientific community, will be described and discussed. These examples highlight the value of HPC infrastructure for a broad range of LSST science cases and the astrophysical community.
Full text
Supercomputer 'Bura' as a Software processing centre for Rubin-related science Tomislav Jurkić Faculty of Physics University of Rijeka LSST@Europe7 | Poznan, Poland | 15-19 September, 2025
Croatian Participation Group @ LSST (CRO-RBI in-kind contribution) 1. Institute Ruđer Bošković, Zagreb (group leader Lovro Palaversa) 2. University of Zagreb (Hvar observatory) 3. University of Rijeka (Faculty of Physics) In-kind contribution •Software development (RBI, UniZg-HO) •Software processing centre (UniRi – HPC 'Bura') LSST@Europe7 | Poznan, Poland | 15-19 September, 2025
Scientific collaborations CPG is active in (but not exclusively focused to): 1. Transients and variable stars (TVS) 2. Stars, Milky way and local volume (SMWLV) 3. Solar System (SSSC) Computer resources in-kind contribution IDAC (International Data Access Centre) SPC (Software processing centre) -Dedicated computer resources for CPU(GPU)-intensive analysis & processing -Demands for CPU(GPU) power, lower disk storage → HPC (high-performance computing) facility LSST@Europe7 | Poznan, Poland | 15-19 September, 2025
Software processing centre HPC 'Bura' at University of Rijeka as SPC: 1. Contribution with substantial amount of CPU-hours for (mainly) stellar time-domain astrophysics 2. Cooperation with Slovenian lite IDAC in Maribor 3. Astrophysicist domain support 4. Analysis of large amount of data + small-scale computing of individual groups 5. Not suited for development (no Jupyter notebooks) LSST@Europe7 | Poznan, Poland | 15-19 September, 2025
Glossary & Acronyms High performance computing facility → top 500 Rmax: 233.565 TFlop/s Rpeak: 287.539 TFlop/s Peak Power (kW): 108.48 Processor: Xeon E5-2690v3 (12 cores @ 2.6 GHz) Cores per Node: 24 Nodes: 288 "Bura" supercomputer 440 175 48
Schematic Architecture 4th Regional LSST Workshop | August 25 – 27, 2025
Cluster (compute nodes) •288 nodes, 2 x Xeon E5-2690 (12c 2.6 GHz)/node, 24 cores/node → 6912 cores •64 GB memory/node, 320 GB disk space/node → 18 TB total memory, 95 TB total disk space Grid computing (Array jobs) •40-50 nodes, 960-1200 cores •Ideal for calculating simulations and/or independent models with large grid of parameters SMP (2 nodes) 16 x Xeon E7-8867 (16c 2.5 GHz)/node → 512 cores, 12 TB total memory, 245 TB total disk space HPC resources @ Bura LSST@Europe7 | Poznan, Poland | 15-19 September, 2025
GPGPU (4 accelerator nodes) Each node: 2 x Xeon E5-2650 CPUs (8c 2.6 GHz) + 2 x Nvidia TeslaK40, 64 GB memory, 320 GB disk space OS Redhat Linux + Slurm Workload Manager Data centre: 1 PB (Lustre scratch file system) Archive: 2.5 PB (tape library) Disk storage extension through regional LSST cooperation grant (Heising-Simons Foundation, 'Preparing for Astrophysics with LSST Program') LSST@Europe7 | Poznan, Poland | 15-19 September, 2025
Reported computational needs for LSST-related science (2023 survey for IDAC workshop, only active members of science collaborations): •100 M CPU-hours in total •35 M CPU-hours for transients & variable stars •35 PB-years disk storage in total •12 PB-years disk storage for transients & variable stars •expected to rise by an order of magnitude LSST-DA survey computer resources and training are top priorities for survey participants Science cases (possible and running) LSST@Europe7 | Poznan, Poland | 15-19 September, 2025
How to get access? Member of Rubin/LSST in-kind contribution What you need to access HPC Bura? •Access is allowed through secured encrypted VPN channel and SSH terminal •Access is also possible through the web, but not reccommended for more demanding work Access to ‘Bura’ 4th Regional LSST Workshop | August 25 – 27, 2025
VPN client Fortieclient VPN (free –MacOS, Linux, Windows) https://www.fortinet.com/support/product-downloads App Store (MacOS) VPN connection bura.uniri.hr (port 443) username and password is the same as when you later connect by SSH Access and computation 4th Regional LSST Workshop | August 25 – 27, 2025
SSH HPC Bura has two login nodes: 172.16.55.121 172.16.55.122 At login node, you can assign jobs at HPC cores by use of Slurm workload manager: https://slurm.schedmd.com/overview.html Possibility to have individual access and scratch folders, as well as to have shared folder accessible from more individual accounts Access and computation 4th Regional LSST Workshop | August 25 – 27, 2025
Web access https://bura.uniri.hr Access and computation 4th Regional LSST Workshop | August 25 – 27, 2025
Guidelines on Bura access on different platforms (MacOS, Linux distributions, Windows) Video tutorials on Bura access, basic use of Slurm workload manager, CUDA, parallel computing: https://hipowered.uniri.hr Tutorials https://cnrm.uniri.hr/tutorials/ Handbook on HPC computing on Bura https://cnrm.uniri.hr/upload/2023/03/hipowered_book.pdf Access and training LSST@Europe7 | Poznan, Poland | 15-19 September, 2025
Introductory HPC InterPython Workshop •Organised by Alex Razim and TVS software task force (including Andjelka Kovacevic) •Online, August 18 – 19, 2025 •Guest lecturer: Gordan Janeš, head of HPC ‘Bura’ (access and Slurm) •21 participants, 8 instructors Follow-up workshop focused on Bura (one day, Q&A open session) planned for November Training LSST@Europe7 | Poznan, Poland | 15-19 September, 2025
HPC ‘Bura’ is planned to host object table (50-75 TB) and offer 2.5 M CPU-hours/year for LSST related computations -hosting stellar catalogue (around 35 TB) What else is expected from the community? What data do we plan to offer? LSST@Europe7 | Poznan, Poland | 15-19 September, 2025
•integration of HPC as SPC with other services, catalogues and databases: HPC is not suited for fetching data outside its own ‘universe’ → no easy access to the outside world •IDAC/SPC cybersecurity •access to the data: ○bulk download after the release, ○use of Rucio software framework for data distribution •local hosting of the part of the Rubin/LSST data (e.g. stellar sources only): how such catalogues will be built? → ‘flagging’ of the sources (extendedness) •customer support (access and use of the facility) •funding (electricity cost) •hardware equipment ‘getting old’ Challenges LSST@Europe7 | Poznan, Poland | 15-19 September, 2025
•Present and upcoming surveys such as LSST will require substantial amount of computing power for TB(PB)-scale of data •HPC ‘Bura’ can be used for computer-intensive calculations and analysis that require large CPU power Takeaway LSST@Europe7 | Poznan, Poland | 15-19 September, 2025
Thank you for your attention! Tomislav Jurkić [email protected] University of Rijeka Croatia LSST@Europe7 | Poznan, Poland | 15-19 September, 2025