Issues Related to Metadata and Metadata Standards Identified while Developing the HAPI Specification and Using HAPI Data

Issues Related to Metadata and Metadata Standards Identified while Developing the HAPI Specification and Using HAPI Data The HAPI specification (Heliophysics Application Programmers Interface) is a specification for the transmission of data and key, but minimal, metadata. The specification developers have intentionally avoided developing metadata standards that significantly overlap with existing metadata standards related to search. In this presentation, I discuss my experience as both a developer of the standard and a user of data served using the standard. I note several issues related to (1) documentation and metadata, (2) where standards are needed, and (3) existing standards. These issues are beyond the scope of the current HAPI project but could be addressed by IHDEA. Bob Weigel, George Mason University 2025 IHDEA Meeting, San Antonio Texas, 2025-10-22 Outline 1. Documentation and Metadata Issues 2. Existing Standards 3. Needed Standards Objective of Presentation ●Identify issues that could be addressed by the IHDEA community. ●It intended to highlight issues and start a discussion of how they could be addressed. ●This is not intended to be a criticism of any single project. Most of the issues have a reasonable explanation (lack of support, out-of-scope, etc.). Common Causes of Problems ●Nobody has complained about it, so must not be a problem. ●Perfection is the enemy of progress. There can be a significant technical debt that accumulates by adding imperfect layers onto imperfect layers. Consider rescoping. (1) Documentation and Metadata Issues (1a) Documentation Issues Many metadata issues encountered in space physics and beyond: ●Description insufficient or missing. ●Description does not point to document that describes; if it does, link is broken. ●Documentation does not have a DOI. Documentation is not versioned. ●Provenance is not given. ●No explanation of relationship to other datasets (e.g., preliminary data should mention or point to definitive data). (1a) Documentation Issues Example Metadata issues with magnetometer providers (WDC, INTERMAGNET, SuperMAG) ●In principle data are simple: observatory, vector, etc. ●All were receptive to suggestions. In some cases I wrote “do you mean …” and this ended up in metadata. ●Nudged into direction of using common HAPI conventions (e.g., name/label: Field_Vector/X,Y,Z) (1a) Documentation Issues Summary ●Primary problem is how metadata gets created. The process I often see is software engineer is asked to do it and then there is no usability review by scientists. Then if nobody complains, it must be good! ●We need a review process where humans check that comprehensive, correct, consistent, and clear. ●Or better feedback mechanism? ●Need focused activity on normalization (do all instruments on spacecraft have same observed region, are keywords for same quantities the same, etc.) (1a) Documentation Issues Question Can/should IHDEA attempt to address?