Interoperable Heliophysics Data Access via HAPI
Abstract
The Heliophysics Application Programmer’s Interface (HAPI) is a simple RESTful API for streaming data that provides a standardized mechanism for accessing distributed time-series datasets, hiding file storage details while offering rich metadata sufficient for automatic plotting and simplified multi-dataset analysis. It was recommended as a standard by COSPAR in 2018, and is in line to become an IHDEA standard. HAPI has been widely adopted by major Heliophysics data providers worldwide, including NASA (SPDF / CDAWeb and CCMC both at GSFC), NOAA (for internal use at SWPC), ESA (at ESAC in Madrid), ViRES for the SWARM mission, and KNMI (Netherlands). HAPI supports data analysis via client libraries in Python, IDL, MATLAB, Java, as well as tools such as Autoplot and PySPEDAS and SPEDAS. HAPI 3.3 was recently released and has enhanced support for metadata related to Open Science. Upcoming HAPI extensions will enable explicit linking of related datasets (such as different cadences), allowing clients to automatically discover and integrate complementary time-series for enhanced visualization, cross-comparison, and contextual analysis. https://hapi-server.org
Full text
Interoperable Heliophysics Data Access via HAPI Jon Vandegriff (JHU/APL) Robert Weigel (George Mason University) Jeremy Faden (Cottage Systems) Sandy Antunes (JHU/APL) Robert Candey (GSFC) Eelco Doornbos (KNMI) • What is HAPI (1 min version) • Latest updates for version the HAPI 3.3 Specification • Usefulness of HAPI in the Science Data System landscape • HAPI and Heliophysics data standards in general • What new capabilities can / should be built on top of HAPI? Outline 10.5281/zenodo.17387480
20 October 2025; San Antonio, TX DASH 2025 2 What is HAPI? In 1 minute.
20 October 2025; San Antonio, TX DASH 2025 3 HAPI is: a data access specification for time-series data https://github.com/hapi-server/data-specification data Server Client Request and Response Standard
20 October 2025; San Antonio, TX DASH 2025 5 HAPI is: a way to hide data storage details behind a simple access API messy data simple API Server Client
20 October 2025; San Antonio, TX DASH 2025 6 HAPI is: a widely adopted standard in Heliophysics *
20 October 2025; San Antonio, TX DASH 2025 7 *Heliophysics in the expansive sense used by DASH: •ground-based measurements (mag, radars, aurora) • upper atmosphere • ionosphere • magnetosphere • the sun • heliosphere • planetary plasma environments • space weather • modeling of any of the above
20 October 2025; San Antonio, TX DASH 2025 8 HAPI is: a widely adopted standard in Heliophysics
20 October 2025; San Antonio, TX DASH 2025 9 HAPI is: a widely adopted standard in Heliophysics • official COSPAR standard for space weather data • in use at multiple Heliophysics institutions •now being adopted at Madrigal
20 October 2025; San Antonio, TX DASH 2025 10 HAPI is: a widely adopted standard in Heliophysics • official COSPAR standard for space weather data • in use at multiple Heliophysics institutions •now being adopted at Madrigal
HAPI 3.3 – fully backwards compatible with 3.2 • added 2 optional attributes to /info response about where measurements were made -location and geoLocation •altitude quantity now allowed as vector component types (common for geo-location) To Support Open Science: • now have distinct serverCitation in about endpoint and datasetCitation in info/ endpoint, and plain citation is deprecated •added resourceID in /about and provenance and licenseURL in /info so that HAPI can describe FAIR data • added a section and appendix on how FAIR principles map to HAPI Units • added new units schema for VOUnits to enumerated list of allowed schemas 20 October 2025; San Antonio, TXDASH 2025 17
Minor fixes / clarifications in the 3.2 HAPI spec • clarified how extra trailing slash should be handled ([#248]#248) • clarified how to name a custom (i.e., non-standard) endpoint (#245) • clarified how scalar parameters can also contain vector components (#244) • clarified requirements for which of bin centers and ranges need to be present (#237) • clarified that HAPI is RESTful rather than strictly based on the original REST concept (#236) • clarified that any non-standard data format in the outputFormats of the /capabilities endpoint needs to begin with x_ (#222) • clarified the difference between title (a short label) in /catalog endpoint versus the description (few lines of details) in /info endpoint (#221) • clarified expectations for id and title in /about endpoint (acronyms ok in id, but expand in title; don't include word HAPI) (#219) • fixed some typos and inconsistencies (#241) • rearranged info section for clarity (#247) 20 October 2025; San Antonio, TXDASH 2025 18
Minor fixes / clarifications in the 3.2 HAPI spec • clarified how extra trailing slash should be handled ([#248]#248) • clarified how to name a custom (i.e., non-standard) endpoint (#245) • clarified how scalar parameters can also contain vector components (#244) • clarified requirements for which of bin centers and ranges need to be present (#237) • clarified that HAPI is RESTful rather than strictly based on the original REST concept (#236) • clarified that any non-standard data format in the outputFormats of the capabilities endpoint needs to begin with x_ (#222) • clarified the difference between title (a short label) in /catalog endpoint versus the description (few lines of details) in /info endpoint (#221) • clarified expectations for id and title in /about endpoint (acronyms ok in id, but expand in title; don't include word HAPI) (#219) • fixed some typos and inconsistencies (#241) • rearranged info section for clarity (#247) 20 October 2025; San Antonio, TXDASH 2025 19 Yes, we are still actively maintaining the spec and responding to questions.
Usefulness of HAPI in the Science Data System Landscape 20 October 2025; San Antonio, TXDASH 2025 20 https://zenodo.org/records/14606063 Robbertz, et al, Fall AGU 2024 HAPI can be used for science access to the data
HAPI also encourages you to standardize your data • don’t do anything weird with your time values • use good variable names • include descriptions • represent vector quantities (magnetic field) as arrays, not sets of scalars • indicate what the FILL value is 20 October 2025; San Antonio, TXDASH 2025 21
HAPI and Heliophysics data standards in general •Units • Coordinate Systems / Frames •maybe Provenance (these exist but are complex) 20 October 2025; San Antonio, TXDASH 2025 22 To make data fully machine interpretable, HAPI needs other standards that don’t exist yet!
What new capabilities can / should be built on top of HAPI? •HAPI to SPASE adapter (see Madrigal talk later today!) -convert HAPI metadata to a SPASE record for a dataset -This could give you guaranteed-correct SPASE that can be automatically kept in sync with the data! •HAPI Data Amalgamator -combine, fuse, average, interpolate any HAPI data content form any server -this would be hugely useful -someone should propose to make this – it is a very fundable idea!! •HAPI for Zenodo – regularized access to all the “stuff” people jam into Zenodo (or similar) -has been proposed once and did not win – talk to me if you want to try this! 20 October 2025; San Antonio, TXDASH 2025 23
20 October 2025; San Antonio, TX DASH 2025 24 Catch this talk later today!!
What new capabilities should be built on top of HAPI? (part 2) • HAPI as a way to regularize other time series lists -file listings -good time intervals (a list of “good data is present” time windows) -event lists •Model-data comparison engine -some elements exist – generalization would be hard • Semantically knowable data structures for measurement types -what if every magnetometer dataset you accessed came into the same data structure in your code? 20 October 2025; San Antonio, TXDASH 2025 25
What new capabilities should be built on top of HAPI? (part 3) • Machine-readable, scientifically useful connections between datasets, i.e., easily use / compare / combine datasets related by: -cadence (same measurements, but different averaging intervals) -coordinate frame (data transformed to other coord sys) -background subtracted version (or other calibration type) -mirror or copy of the data 20 October 2025; San Antonio, TXDASH 2025 26 Timeline Viewer at KNMI is client that auto-switches to correct cadence dataset based on time window size in current view. We are generalizing this clever addtion!