Towards FAIR for AI: Quality in Heterogeneous Digital Objects

Towards FAIR for AI: Quality in Heterogeneous Digital Objects Daniel Garijo, Ontology Engineering Group, Universidad Politécnica de Madrid, Spain FAIR&AI Symposium November 27th, 2025 [email protected] @dgarijov FAIR&AI Symposium. TU Graz. November 27th, 2025 Background 2 https://www.force11.org/group/fairgroup/fairprinciples https://doi.org/10.1038/sdata.2016.18 2016 FAIR&AI Symposium. TU Graz. November 27th, 2025 Overview 3 -The key motivations for FAIR -The multiple extensions of FAIR -FAIR for AI: Initiatives and challenges FAIR&AI Symposium. TU Graz. November 27th, 2025 4 Is FAIR the goal, or a means to an end? How did we get here? A brief walk down history lane… FAIR&AI Symposium. TU Graz. November 27th, 2025 Why FAIR? - Improving scientific credit 20252010 20122013 https://sfdora.org/read/ 5 FAIR&AI Symposium. TU Graz. November 27th, 2025 DORA: metrics based on articles are not enough https://sfdora.org/read/ 6 FAIR&AI Symposium. TU Graz. November 27th, 2025 Other open letters: Leiden Manifesto, Barcelona declaration 7 2013 2024 FAIR&AI Symposium. TU Graz. November 27th, 2025 Why FAIR? - Improving reproducibility in Science 8 https://www.nature.com/articles/533452a Adapted from: Yolanda Gil. The Scientific Paper of the Future" training materials for the OntoSoft project are available from http://doi.org/10.5281/zenodo.159206 FAIR&AI Symposium. TU Graz. November 27th, 2025 Background 9 https://www.force11.org/group/fairgroup/fairprinciples https://doi.org/10.1038/sdata.2016.18 2016 FAIR&AI Symposium. TU Graz. November 27th, 2025 A motivating example: CPV classifier 16 Let’s create a classifier that takes public tender (contract) descriptions to predict their corresponding CPV. How to find training data? -All datasets are available and have metadata, API access :) - Difficult to find dataset variables associated with my search - Many datasets do not contain the data I need FAIR&AI Symposium. TU Graz. November 27th, 2025 CPV classifier: Once data is found… 17 Challenges: - Dataset needs to be manually inspected to check the columns I need - Data may be multilingual -Duplicate values -Incomplete values -Inconsistent annotations (annotation information is usually scarce) FAIR&AI Symposium. TU Graz. November 27th, 2025 Main challenge: quality is key for ML applications 18 Credit to Eduardo Ordax, Christian Krug. https://www.linkedin.com/posts/eordax_ai-data-genai-activity-7275487825238634496-ZRnF/ Metadata of poor quality will yield incorrect results for training AI models Data of poor quality will yield poor AI results, despite its FAIRness FAIR&AI Symposium. TU Graz. November 27th, 2025 FAIR4AI: Existing initiatives 19 Initiatives for dataset metadata Initiatives for software metadata (ML Model) FAIR&AI Symposium. TU Graz. November 27th, 2025 FAIR4AI: Existing initiatives 20 Initiatives for dataset metadata Initiatives for software metadata (ML Model) FAIR&AI Symposium. TU Graz. November 27th, 2025 Dataset metadata standards: W3C DCAT 21 https://www.w3.org/TR/vocab-dcat-3/ Example: https://data.europa.eu W3C standard for dataset and catalog descriptions FAIR&AI Symposium. TU Graz. November 27th, 2025 Dataset metadata standards: Schema.org 22 More than a third of the web is annotated with Schema.org https://datasetsearch.res earch.google.com FAIR&AI Symposium. TU Graz. November 27th, 2025 ML Datasets metadata standard: Croissant ML 23 Schema.org extension designed to describe ML datasets. -Includes descriptions of variables and expected types { "@type": "cr:Field", "name": "age", "description": "The second column contains the age.", "dataType": "sc:Integer", "source": { "fileObject": { "@id": "minimal.csv" }, "extract": { "column": "age" } } } https://github.com/mlcommons/croissant/ FAIR&AI Symposium. TU Graz. November 27th, 2025 Describing variables in detail: I-ADOPT 24 Barbara Magagna, Sirko Schindler, Maria Stoica, Gwenaelle Moncoiffe, Anusuriya Devaraju, Alison Pamment, I-ADOPT Framework ontology, Retrieved from: https://w3id.org/iadopt/ont/1.1.0 FAIR&AI Symposium. TU Graz. November 27th, 2025 FAIR4AI: Existing initiatives 25 Initiatives for dataset metadata Initiatives for software metadata (ML Model) FAIR&AI Symposium. TU Graz. November 27th, 2025 Conclusions 32 FAIR plays a key role in AI: -Metadata -Interoperability -Search: Variable descriptions -Provenance: key to understand how an AI model was created -Interoperability, both at data and metadata level Quality is a remaining open challenge unaddressed by FAIR -Quality of data -Quality of metadata FAIR is a means to an end (reproduce, adopt, etc.) -High quality datasets and models should be FAIR -Remember https://paperswithcode.com ? FAIR&AI Symposium. TU Graz. November 27th, 2025 33 FAIR alone is not the goal FAIR Quality Towards FAIR for AI: Quality in Heterogeneous Digital Objects Daniel Garijo, Ontology Engineering Group, Universidad Politécnica de Madrid, Spain FAIR&AI Symposium November 27th, 2025 [email protected] @dgarijov