scieee AI-readable full text Open interactive document viewer

Describing Data for Discovery: RSEs, Croissant, and the Future of Research Metadata

Deb, Sreeparna

Publisher
Zenodo
DOI
10.5281/zenodo.17967438
Repository
zenodo.org
Language
en

Abstract

Many research projects collect or create datasets as part of their outcomes, but these are often hard to find, interpret, or reuse. A key reason is the lack of structured, machine-readable metadata that can support both human understanding and automated tooling. This creates friction with data discovery, integration, and reuse—especially in machine-learning workflows.This talk makes a case for adopting Croissant, a metadata format developed by MLCommons to address these challenges. It provides a standardised vocabulary and structure to describe datasets in ways that can be validated, shared, and used programmatically. Although Croissant originated from the ML community, its design is broadly applicable to any research data ecosystem. RSEs are in a unique position to help the widespread adoption of such standards. Since we often work with tools and platforms that manage datasets, we can embed good metadata practices by design, not as an afterthought.The talk elaborates on what it looks like in practice to integrate Croissant into a real-world project workflow. To this end, we draw on EnergySHR, a project supporting AI for energy and sustainability. It aims to help researchers share energy-related datasets and references. During submission, users provide metadata to generate structured, machine-readable descriptions of each dataset. These are validated and embedded into the dataset's page. This helps support structured web search and enables automated loading of datasets in downstream workflows.By embedding standards like Croissant into the tools we build and support, RSEs can make research outputs more visible and reusable. These are not just technical improvements but key contributions to research excellence.A recording of this session is available on YouTube: https://youtu.be/h0M66R7Oa80

Full text

Describing Data for Discovery with Croissant Sreeparna Deb, Research Software Engineer @ REIT TU Delft ML runs on data — But reuse is still painful 01 Friction shows up as… Most datasets have unique layouts Combining datasets is cumbersome Metadata is technically shared but it’s not really reusable Requires writing custom loading code 01 02 03 04 What is the bottleneck? 02 Lack of consistent Metadata What we need We need metadata that helps not just describe a dataset, but actually use it — across frameworks, pipelines, and platforms. What we have Existing vocabularies like DCAT and schema.org exist, but they weren’t built with ML workflows in mind. They tell you that a dataset exists — not how to load, interpret it. Croissant – metadata for ML ready datasets 03 What is in a croissant file? Metadata Layer High level dataset related metadata based on schema.org/Dataset vocabulary Structure Layer Specifies how data is organized and accessed within files using RecordSet, enabling unified loading of structured and unstructured data Resource Layer Describes the actual data files (e.g., CSVs, archives) using FileObject and FileSet, supporting complex file layouts Semantic Layer Adds ML-specific meaning like train/test splits, labels, and annotations 02 04 03 01 What is in a croissant file? Metadata High level dataset related metadata based on schema.org/Dataset vocabulary Structure Specifies how data is organized and accessed within files using RecordSet, enabling unified loading of structured and unstructured data. Resource Describes the actual data files (e.g., CSVs, archives) using FileObject and FileSet, supporting complex file layouts common in ML datasets. Semantic Adds ML-specific meaning like train/test splits, labels, and annotations 02 04 03 01 Metadata Layer What is in a croissant file? Metadata High level dataset related metadata based on schema.org/Dataset vocabulary Structure Specifies how data is organized and accessed within files using RecordSet, enabling unified loading of structured and unstructured data. Resource Describes the actual data files (e.g., CSVs, archives) using FileObject and FileSet, supporting complex file layouts common in ML datasets. Semantic Adds ML-specific meaning like train/test splits, labels, and annotations 02 04 03 01 Resource Layer Thank you! CREDITS: This presentation template was created by Slidesgo, and includes icons by Flaticon Resources For the slide content ●ESIP YT video on introducing Croissant ●MLC Croissant paper Code ●MLC croissant source code ●Croissant working example