NeXusCreator — Standardizing Science, Simplifying Data

NeXusCreator — Standardizing Science, Simplifying Data Turn raw data into NeXus/HDF5 — clean, consistent, reusable Authors: H. Perez Ponce1, R. Krahl1, D. Tomecki1, W. Smith1, P. Braun1, H. Görzig1 1Helmholtz-Zentrum Berlin für Materialien und Energie, Hahn-Meitner-Platz 1, 14109, Berlin Contact: [email protected] Introduction NeXus is the community standard for neutron, Xray and muon data, created to enable interoperable analysis and visualization across facilities. NeXusCreator makes producing compliant NeXus/HDF5 straightforward, even in heterogeneous environments. Motivation At BESSY II (HZB), beamlines output data in many formats ( SPEC , TIFF , DAT , HDF5 , ...). This heterogeneity: • Slows down analysis • Complicates data sharing • Hinders the FAIR principles (Findable, Accessible, Interoperable, Reusable) Adoption: many modern beamlines write NeXus/HDF5 as the primary format from day one. NeXusCreator Approach NeXusCreator tackles these challenges by: • Standardizing outputs: transform raw files + template into valid NeXus ( .nxs ) files • Lowering barriers: no coding required; usable from CLI or Python • Enabling FAIR practices: interoperability and reusability across beamlines • Adapting to diverse sources: supports SPEC , DAT , TIFF , HDF5/NeXus , SIF , XAS , and plugins What it does NeXusCreator converts raw files into portable, FAIRcompliant NeXus datasets. • Generate templates: lightweight NeXus definitions ( .nxd / .yml ) • Work at scale: batch-process folders with recursion and filters • Add context: attach units & descriptions via CSV ; guided by NXDL if desired • Re-use data: extract or link from existing NeXus/HDF5 files • Use anywhere: simple CLI, Python API, or plugins Why it matters • Consistent archiving  data stays usable longterm • Collaboration made simple  share without reformatting • FAIR by design  Findable, Accessible, Interoperable, Reusable • FAIR in practice: machine-actionable interoperability and reuse • Accelerates science  less time fixing files, more time discovering • Open source (Apache 2.0)  community-driven, future-proof Summary Developed at Helmholtz-Zentrum Berlin (HZB), NeXusCreator is an open-source tool that helps facilities and users converge on a common NeXusbased data structure — without changing the original acquisition process. How it works 1) Auto-generate a NeXus Description from raw data. Create a lightweight, human-readable NeXus definition file ( .nxd , ASCII) that mirrors the target structure of your dataset, or use YAML ( .yml ) for a more standard syntax. This provides an editable starting point before generating the final NeXus/HDF5 file. •NXD - NeXus Description file ( .nxd ) nexuscreator -g template.nxd -i data.spec # template.nxd entry: @NX_class = NXentry scan_{num}: @NX_class = NXgroup instrument: @NX_class = NXinstrument logs: energy: @NX_class = NXlog value: NX_FLOAT64[] = scan{num}_energy h: @NX_class = NXlog value:NX_FLOAT64[] = scan{num}_h •YAML - NeXus Description file ( .yml ) nexuscreator -g template.yaml -i data.spec yaml # template.yml entry: attributes: NX_class: "NXentry" scan_{num}: attributes: NX_class: "NXgroup" instrument: attributes: NX_class: "NXinstrument" logs: energy: attributes: NX_class: "NXlog" value: dtype: "NX_FLOAT64[]" value: scan{num}_energy ... During conversion, scan_{num} expands to scan_01 , scan_02 , ... (Optional) If editing the NeXus Definition is not required, this step can be hidden to the user by using auto-generate-nxd , which automatically creates the ( .nxd ) and use it to generate the final ( .nxs ) NeXus/HDF5 file. 2) Convert the NeXus Definition ( .nxd or .yml ) to NeXus ( .nxs ) nexuscreator -n template.nxd -i data.spec -o out.nxs or nexuscreator -n template.yml -i data.spec -o out.nxs # out.nxs (HDF5 NeXus structure) /entry /scan_01 /instrument /logs /energy value (float64[...]) /h value (float64[...]) Why HDF5? Self-describing structure, chunked I/O and built-in compression enable fast, portable and scalable data. SchemaPlacer SchemaPlacer is an NXDL indexer built into NeXusCreator that reads NeXus data schemas (application definitions) and proposes canonical NeXus paths. What it does • Indexes NXDL data schemas: collects required fields/paths from application definitions and base classes • Matches names: normalizes dataset names, applies aliases (e.g., hv → energy ) • Scores candidates: selects best fit from application definition • Builds objects: generates NeXus dictionaries with inferred @dtype and optional @units Usage example: (NXDL guidance + data schema (application definition) insertion + dataset definitions) nexuscreator -g template.nxd -i data.spec -nxdl-root ./nexus_definitions -app-def NXxas -metadata-csv dataset_definitions.csv # template.nxd entry: @NX_class = NXentry scan_{num}: @NX_class = NXgroup instrument: @NX_class = NXinstrument monochromator: @NX_class = NXmonochromator energy:NX_FLOAT64[] = scan{num}_energy @units = "eV" @description = "photon energy (monochromator)" sample: @NX_class = NXsample h:NX_FLOAT64[] = scan{num}_h @units = "rlu" @description = "Reciprocal space coordinate" XAS: @NX_class = NXsubentry name:NX_CHAR = "NXxas" instrument: @NX_class = NXinstrument source: @NX_class = NXsource type:NX_CHAR = type name:NX_CHAR = name probe:NX_CHAR = probe monochromator: @NX_class = NXmonochromator energy:NX_FLOAT64[] --> /entry/scan_{num}/ instrument/ monochromator/energy ... SchemaPlacer automatically maps each quantity into the right NeXus location: • Energy → stored in the NXmonochromator (instrument), because it describes the photon energy set by the beamline • h, k, l → stored in the NXsample, because they define the position of the sample in reciprocal space The data schema (application definition) is extracted from the official NeXus definitions and inserted at the correct place inside the NeXus file, ensuring semantic correctness. Raw Data DAT, HDF5, ... Data Schema NeXus Definitions NeXus (.nxs) FAIR-compliant Links NeXusCreator Code & Documentation NeXus Standard Official website