Heterogeneous Distributed Data Management in Academia

US-RSE 2025 Heterogeneous Distributed Data Management in Academia Josh Borrow (University of Pennsylvania & Simons Observatory) 500 people in the collaboration! Total project cost: around $200m 500 TB - 1 PB raw data a year Differing hardware landscapes Sales Research Tracking Experiment National Facility Core Analysis Additional Analysis Time-Sensitive Events Data Release Publication Web Services Local Clusters On-site Compute and Storage Data Warehouse Analysis Web Services Dashboard Reports Egress Performed on owned or configurable hardware Performed on owned or configurable hardware Differing hardware landscapes Sales Research Tracking Experiment National Facility Core Analysis Additional Analysis Time-Sensitive Events Data Release Publication Web Services Local Clusters On-site Compute and Storage Data Warehouse Analysis Web Services Dashboard Reports Egress Performed on owned or configurable hardware Performed on owned or configurable hardware Site UCSD NERSC Princeton ✈ ✈ ✈ S n e a k e r N e t T r a n s f e r ( H a n d c a r r y ) Data Destinations Main node: national facility (NERSC) on U.S. West Coast.! Copies need to be sent to Princeton and the U.K., with more nodes possible. Academic workflows Experiment Data Shared Filesystem Batch Job Processing Secondary Data Artifact Visualize Unlinked Database Share Path on Filesystem scp? X forward? Data found at pre-known path Data requirements •Replication of a POSIX filesystem structure on high performance disks all data-centers.! •Custom retention policy control on a per-center basis.! •Low latency (ideally sub-hour).! •Ability to work in ‘sneakernet’ mode, where disks are handcarried for data transfer. Software setup •A custom data transfer orchestrator: ‘Librarian’ uses globus for inter-node transfer. Each site controls their own data policies, and are only responsible for responding to questions about what data they hold.! •Full data cataloging in postgres, available through a HTTP client.! •Data integrity is verified continually through inter-node communication. System setups NERSC Rancher Princeton Podman on dedicated VM SO:UK Podman NERSC managed endpoint Princeton computing endpoint Custom endpoint J e r e m y M y e r s U n i v e r s i t y o f P e n n s y l v a n i a D i v e s h J a i n U n i v e r s i t y o f M a n c h e s t e r G i a n n i s P a r a s k e v a k o s P r i n c e t o n R e s e a r c h C o m p u t i n g Visualizing spatial data in the browser •At the core of our project are giant astronomical images (think RAW files), each with up to 900 megapixels.! •Pain point: visualizing these; users may have 50-100 of such images to ‘flip’ through (250-500 GB of data). J e r e m y M y e r s U n i v e r s i t y o f P e n n s y l v a n i a D i v e s h J a i n U n i v e r s i t y o f M a n c h e s t e r G i a n n i s P a r a s k e v a k o s P r i n c e t o n R e s e a r c h C o m p u t i n g Lessons from jupyter •It has become common for users to forward ports from HPC machines to their laptops for e.g. Jupyter services.! •With this in mind, we built a command-line driven web application for visualizing these maps.! •The application relies heavily on just-in-time data processing to keep memory footprint low. Composable software •By building ‘tilemaker’ as a composable library, we are able to have a deployed version at maps.simonsobservatory.org! •Like hippo, we have group-based access control, with some data public and some not.! •The map viewer is a highly useful tool for data releases. Takeaways •When dealing with heterogeneous computing environments, having local experts is critical. •Decentralization is not always the best strategy, and it’s worth pushing back against. •Web-based tools driven by CLI interfaces provide powerful experiences for single users, and provide extra value as ‘hosted versions’ for public access. •We build everything in the open (github.com/simonsobs) but our stuff is all very early-stage, as we’re a young experiment. J e r e m y M y e r s U n i v e r s i t y o f P e n n s y l v a n i a D i v e s h J a i n U n i v e r s i t y o f M a n c h e s t e r G i a n n i s P a r a s k e v a k o s P r i n c e t o n R e s e a r c h C o m p u t i n g