NeXus and Electron Microscopy

NeXus and Electron Microscopy Federica Bazzocchi, Istituto di Ricerca per l’Innovazione Tecnologica (RIT) Trieste Optical, Electron, and Scanning Probe Microscopy Online Workshop 6.11.2024 OUTLINE ❑Brief recap on FAIR principles ❑NeXus file format ❑Nxem ❑Final remarks Foto di Brian Penny da Pixabay ❑Brief recap on FAIR principles Door SangyaPundir - Eigen werk, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=53414062 https://doi.org/10.1038/sdata.2016.18 F F1: (Meta) data are assigned globally unique and persistent identifiers F2: Data are described with rich metadata F3: Metadata clearly and explicitly include the identifier of the data they describe F4: (Meta)data are registered or indexed in a searchable resource A A1: (Meta)data are retrievable by their identifier using a standardized communication protocol A1.1: The protocol is open, free and universally implementable A1.2: The protocol allows for an authentication and authorization procedure where necessary A2: Metadata should be accessible even when the data is no longer available I R I1: (Meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation I2: (Meta)data use vocabularies that follow the FAIR principles I3: (Meta)data include qualified references to other (meta)data R1: (Meta)data are richly described with a plurality of accurate and relevant attributes R1.1: (Meta)data are released with a clear and accessible data usage license R1.2: (Meta)data are associated with detailed provenance R1.3: (Meta)data meet domain-relevant community standards ❑Brief recap on FAIR principles: the metadata role Structural Interoperability: Data have to use the same model representation to be processed by different app/algorithm Semantic Interoperability: Data have to use common vocabulaties in order to be understood in their meaning both by humans and not humans Technical Interoperability: Data have to be understood by different app, exchanged by common protocols I of FAIR ❑Brief recap on FAIR principles: the interoperability challenge Slides by Elena Giglia DOI 10.5281/zenodo.6761944 Managing data properly is in the primary interest of any researcher, as the whole research process results streamlined and more effective FAIRness VS OPENness MANAGED FAIR 1. DATA SHOULD BE AS OPEN AS POSSIBLE 2. BUT IF DATA ARE NOT «FAIR», OPENING IS RISKY (MISUSE, MISINTERPRETATION, …) 3. IF DATA ARE NOT PROPERLY MANAGED FROM THE BEGINNING, IT’S ALMOST IMPOSSIBLE TO MAKE THEM «FAIR» [WITH EOSC MANAGED/FAIR INCREASINGLY OVERLAPPING, «FAIR-BY-DESIGN»] FAIR ≠ OPEN In Horizon Europe data should be «as open as possible and as closed as necessary» OPEN Metadata + interoperability = Standard format A GOOD CHOICE IS GIVEN BY THE NEXUS DATA FORMAT NeXus data format NeXus is a common data exchange format for neutron, X-ray, and muon experiments. NeXus is built on top of the scientific data format HDF5 and adds domain-specific rules for organizing data within HDF5 files in addition to a dictionary of well-defined domain-specific field names. Application definition are supervised/regulated by NIAC The NeXus data format has three purposes: •Raw data: NeXus defines a format that can serve as a container for all relevant data associated with a scientific instrument or beamline. •Processed data: NeXus also defines standards for processed data. This is data which has underwent some form of data reduction or data analysis. NeXus allows storing the results of such processing together with documentation about how the processed data was generated. •Standards: NeXus defines standards in the form of application definitions for the exchange of data between applications. NeXus provides standards for both raw and processed data. What is HDF5? Hierarchical Data Format Let’s put everything together… An example: Nxem (contributed definition Partner consortia in the German National Research Data Infrastructure are here e.g. NFDI-MatWerk, NFDI4Ing, NFDI-BioImage, NFDIMicrobiota, NFDI4Health, and e.g. NFDI-Neuro) NXem is a NeXus application definition for the normalized representation of electron microscopy research. It is an extension of the Nxem_base base class. This application definition is thus an example of a general description with which to normalize specific pieces of information and data collected within electron microscopy research. This application definition is also a blueprint which shows how users can build specific application definitions by reusing em-specific base classes - and thus represent electronmicroscopy-specific content. NXem <?xml version='1.0' encoding='UTF-8'?> <?xml-stylesheet type="text/xsl" href="nxdlformat.xsl"?> <definition xmlns="http://definition.nexusformat.org/nxdl/3.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" category="application" type="group" name="NXem" extends="NXobject" xsi:schemaLocation="http://definition.nexusformat.org/nxdl/3.1 ../nxdl.xsd"> NXem Base Classes NXem Base Classes Nxentry profiling citeID Definition (version, value=Nxe m) Experiment identifier groups fields Exp alias Exp desc Start_ time End time userID serializedI D Coordinate system set measurement simulation roiIDsample optional suggested required NXem NXsample Type Value=exp eriment/si mulation Identifier groups fields Atom types Thickness (units) Preparati on date Name Parent identifier Density (units) description optional suggested required NXem •Nxarray: this library take arrays as inputs and returns nexus •Pynxtool: library that extends NeXus for experiments and characterization in Materials Science and Materials Engineering and serve as a NOMAD parser implementation for NeXus. •Nexus-format: This package provides a Python API to open, create, and manipulate NeXus data written in the HDF5 format. The 'nexusformat' package provides the underlying API for NeXpy, which provides a GUI interface for visualizing and analyzing NeXus data. •Python-nexus:python-nexus provides simple nexus file-format reading/writing tools, and a small collection of nexus manipulation scripts. NeXus: python libraries Own scripts FINAL REMARKS ❑To be FAIR compliant we need choosing a standard ❑NeXus file format is a reasonable choice because it has behind a huge active community ❑Nxem may describe the majority of EM experiments Thanks for your attention!