Progress on the LSST lite-IDAC@PIC

Progress on the LSST lite-IDAC@PIC J. Carretero, R. Sanfeliu, M. Santamaria, S. Van den Bogart, A. Luelmo, P. Tallada, J. Delgado, F. Torradeflot, E. Planas, I. Sevilla on behalf of the PIC team 2 J.Carretero - LSST@Europe7 - Poznań (Poland) - Sept 15-19, 2025 ●Computing and storage infrastructure ●CosmoHub upgrades ○Virtual Observatory (VO) compliance ●Authentication and Authorization ●DP1 data transfer & ingestion Outline 3 J.Carretero - LSST@Europe7 - Poznań (Poland) - Sept 15-19, 2025 ●Founded in 2003: collaboration between IFAE and CIEMAT ○One of the 14 Tier-1 centers worldwide ○PIC joined the Spanish Supercomputing Network in 2020 ●Team of 26 people (50% scientists - 50% engineers) ○Agile teams that embed in scientific groups to ▪Understand the experiment ▪Follow the evolution of data analysis requirements ▪Develop & prototype tools for data management and analysis ●What we do ○R&D in methodologies and tools for advanced data analysis ▪Lead and participate in R&D projects. Software and Computing WPs ○Operate services for the preservation, analysis and sharing of data ▪Run prod. services for experiments: Euclid, PAUS, CTA, MAGIC, LIGO/Virgo, DUNE and LHC ▪Provide data analysis services for research groups: IFAE, CIEMAT + others Port d’Informació Científica 4 J.Carretero - LSST@Europe7 - Poznań (Poland) - Sept 15-19, 2025 Computing and storage infrastructure ●Computing: ○200 CPU cores ✔ ■Backfilling approach using the PIC Hadoop Cluster ■Public tender in progress for FAT nodes ○One GPU server with 8 GPUs (public tender open) ●Storage: ○500TB of disk ✔ ■HDFS storage capacity (~3PB net in total) ■dcache pools (public tender in progress) ●Magnetic tapes for cold storage 768 CPU cores, 16 TiB RAM, ~3PiB net storage, 60 TB NVMe cache 5 J.Carretero - LSST@Europe7 - Poznań (Poland) - Sept 15-19, 2025 ●CosmoHub as a service to provide massive catalog exploration and distribution ○Run on top of the PIC Hadoop cluster: ■768 CPU cores, 16 TiB RAM, ~3PiB net storage, 60 TB NVMe cache ●New frontend including already some new features: ○Improved workspace ■Improved guided filters ■HEALPiX maps & array plot ●More to come ○new API in development ○VO-protocol and format compliant (TAP, ADQL, UWS, VOTable, VOSpace, …) ○Performance improvements (arrow-based communication protocol) ○Upload your own catalogs (webdav-hdfs door in production) ○Spatial cross-matching (including support for MOC maps and different messengers) upgrades 6 J.Carretero - LSST@Europe7 - Poznań (Poland) - Sept 15-19, 2025 ●Access Control Measures for Rubin DP1 and Pre-Release Data at IDAC@PIC ○Pre-release data protected by a dedicated lsstidac POSIX group (manual access requests) ○DP1 data access via CosmoHub initially requires direct email approval ●Future advanced access will utilize a custom Keycloak extension (PIC-LSST-SPI) to automate and secure user access based on Rubin's data rights: ○Keycloak SPI-Based Authorization Integration for Rubin Data Rights Data access control 7 J.Carretero - LSST@Europe7 - Poznań (Poland) - Sept 15-19, 2025 ●Current A&A platform ○FreeIPA: stores all accounts and allows LDAP integration ○Keycloak: provides SSO and interfaces with external services ●New developments (as Keycloak plugins) ○Automatic (non-privileged) account registration ○Two privilege escalation procedures ■manual approval via administrator upon request ■automatic via OpenId federation (for projects with public IdP, such as LSST, CTAO or Euclid) ○Automatic membership background checks ■project membership is always up to date ■avoids grace periods ●This will enter in production this fall ○Source code available and collaborators are welcomed A&A compliant with LSST In development 8 J.Carretero - LSST@Europe7 - Poznań (Poland) - Sept 15-19, 2025 ●Technological platform ○dcache version: 9.2.25.1 ○supported VOs: atlas, cms, ctao, dune, lhcb, virgo, … ○supported protocols: webdav, xrootd, gridftp ○Enabled features: HTTP-TPC, macaroons, Quotas (Space Tokens) ●Rucio configuration ○RSE Name: IDAC_ESP-BCM ○Local group: group lsstidac ○Local user: lsstidac01 ●DP1 already transferred to PIC ○4.8 TB transferred in <48 hours DP1 data transfer 9 J.Carretero - LSST@Europe7 - Poznań (Poland) - Sept 15-19, 2025 ●10 tables ingested into CosmoHub ○Using Spark as loading / transforming / writing tool ○Apache Hive as data warehouse that powers SQL engine ●Some issues found in our process: ○Neither Spark (< v3.5) nor Hive do not support TIMESTAMP with nanosecond precision ■Converted to microsecond precision ○Hive / CosmoHub do not support TIMESTAMP_NTZ (without timezone) ■Converted to TIMESTAMP (assumed UTC timezone) ○Schema mismatch, as well as minor inconsistencies between documentation, in some parquet files ●Developing an automatic process for ingesting catalogs into CosmoHub DP1 catalogs in 16 J.Carretero - LSST@Europe7 - Poznań (Poland) - Sept 15-19, 2025 ●Facilities, ~120 kW IT ○~80 kW in 150 m2 air-cooled room ■high efficiency, PUE 1.44 ○~40 kW in 25 m2 liquid immersion cooling system ■PUE 1.1 ●Data processing services ○Disk - dCache: 30 PB ○Tape - Enstore: 70 PB ○Computing - HTCondor: 13000 cores, 18 GPUs ○Computing - Hadoop: 768 cores, 3 PB net storage ●Connectivity ○2x100 Gbps to Academic Network ○Largest data mover in Spanish academic network: 100PB in+out per year PIC data center IBM TS4500 171717 J.Carretero - LSST@Europe7 - Poznań (Poland) - Sept 15-19, 2025 What do we do at PIC? ●Data transfer ●User support so they can process and analyze the data ●Store data and/or results ●Data distribution Scientific Results ●Provide tools to enable science ●Promote the dissemination of the science we work with data Experiments Instruments Simulations 181818 J.Carretero - LSST@Europe7 - Poznań (Poland) - Sept 15-19, 2025 What do we do at PIC? ●Data transfer ●User support so they can process and analyze the data ●Store data and/or results ●Data distribution Scientific Results ●Provide tools to enable science ●Promote the dissemination of the science we work with data Experiments Instruments Simulations 19 J.Carretero - LSST@Europe7 - Poznań (Poland) - Sept 15-19, 2025 lite-IDAC@PIC features ●Resources (once completed) ○200 CPU cores, 8 GPU and 500TB of disk ●Services ○Federated Authentication ○Data transfer: Rucio (✔) ○Jupyterlab ✔ ■Notebooks over PIC's HTC cluster ●User-defined session resources: CPUs, GPUs, Memory ■SSH terminal, VNC desktop, VS Code IDE ■Dask / Spark ○Data exploration and distribution (CosmoHub) ✔ ■On top of the PIC Big Data Common Service (based on Hadoop) Shown in LSST@Europe6 - La Palma - Sept 16-20, 2024 20 J.Carretero - LSST@Europe7 - Poznań (Poland) - Sept 15-19, 2025 MM- ●Multi-instrument, -frequency, -messenger ○Electromagnetic ■Optical:DES ✔, Euclid ✔, Gaia ✔, ■Gammas (MAGIC ✔, CTAO/LST ) ○GWs (LIGO/Virgo) ○Neutrinos (ANTARES, KM3NeT) ●Massive transfers: ○Rucio + FTS ✔ ○Rclone ✔ ●Standardize & facilitate access: ○Federated Authentication ■Tokens ○Virtual observatory standards ■VOTable, ADQL, TAP, UWS ○Persistent identifiers (DOIs) ●Advanced features ○Users can upload catalogs ○User Defined Functions (UDFs) ✔ ○Enhanced plotting ■HEALPix maps ■Improved density ●Jupyterlab integration ✔ ○Notebooks over PIC's HTC cluster ○SSH terminal, VNC desktop, VS Code IDE ●Massive algorithms (Dask ✔ / Spark ✔) ○Spatial cross-match ○Light curves & SED ○Synthetic galaxy catalog generation ✔ ●Infrastructure expansion ○Computing / storage ■512 cores, 0.5 PB - disk, 3PB - tapes 21 J.Carretero - LSST@Europe7 - Poznań (Poland) - Sept 15-19, 2025 PIC’s Hadoop distribution: Shepherd ●Motivation ○Avoid vendor lock-in (Cloudera) ○Binaries no longer accessible ○Outdated versions ○Flexibility in the combination of components ●Shepherd Hadoop distribution ○Consolidated in a single docker image ■datanode and nodemanager also outside docker ○Tested and deployed using CI/CD Pipeline ■pseudo-distributed, development and preproduction setup ○Configuration tracked in a git repository ○Additional RPM for client-only installation ○Monitored using JMX protocol Component HDP 3.1.4 Shepherd 1.0.0 Atlas 1.1.0 2.2.0 Hadoop 3.1.1 3.2.3 HBASE 2.0.2 2.5.8 Hive 3.1.0 3.1.2 Kafka 2.0.0 2.5.0 Oozie 4.3.1 5.2.1 Ranger 1.2.0 2.4.0 Solr 7.7.0 8.11.2 Spark 2.3.2 3.4.2 Tez 0.9.1 0.10.1 Zookeeper 3.4.6 3.7.1 Kerberos MIT KDC FreeIPA 22 J.Carretero - LSST@Europe7 - Poznań (Poland) - Sept 15-19, 2025 ●HEALPix (enable spherical analysis) ○Conversion: ang2pix, pix2ang, ang2vec, vec2ang, vec2pix, pix2vec ○Ordering: nest2ring, ring2nest ○Sizing: nside2npix, npix2nside, order2npix ○Other: neighbours, maxpixrad, nside2order ●Array aggregation ○Summary: array_min, array_max, array_count, array_sum ○Dispersion: array_avg, array_stddev_pop, array_stddev_samp array_var_pop, array_var_samp ●Spherical geometry ○Types: point, box, circle, polygon ○Constructors: from coordinates / pixels ○Simple: area, centroid, coord1, coord2, distance ○Spatial relationships: contains, intersects ○Region extension (MOC): complement, intersection, union User Defined Functions 23 J.Carretero - LSST@Europe7 - Poznań (Poland) - Sept 15-19, 2025 ● Webdav door ○ http protocol to read / download files ○ Command line access (upload included) ○ Work with external tools (Rclone) ● Frontend dCacheView ○ User-friendly frontend to upload/download files ○ Share a URL with a temporal token to download files (up to 1 week) Storage (https access) External access to mass storage (https) 24 J.Carretero - LSST@Europe7 - Poznań (Poland) - Sept 15-19, 2025 ●Automatic non-privileged PIC account registration ●Automated project access request management ○manual approval via administrator ○automatic via OpenId federation ●Automatic membership background checks ○project membership is always up to date ○avoids grace periods A&A compliant with LSST In development 25 J.Carretero - LSST@Europe7 - Poznań (Poland) - Sept 15-19, 2025 ●Rucio: comprehensive solution for managing, organizing, and distributing data ●Led an effort to integrate a Rucio deployment in a single helm chart ○Includes a PostgreSQL instance, the Rucio server and the daemons. ○Monitoring using Grafana and ElasticSearch ○Token support (in progress) ●Deployed several instances to manage different project's transfer needs ○MAGIC ○CTAO/LST (in progress) ○ICFO (in progress) ○InCaem (in progress) ○LSST lite-IDAC (evaluating) Massive data transfer: Rucio + FTS