DDI Adoption Metrics – Insights from a KonsortSWD Short Project

DDI Adoption Metrics – Insights from a KonsortSWD Short Project Knut Wenzig EDDI 2025, Budapest, 3 December 2025 doi:10.5281/zenodo.17760658, License: CC BY-SA 4.0 (exceptions see last slide) Abstract DDI Adoption Metrics • Knut Wenzig • EDDI2025 Budapest • 3 December 2025 2 DDI metadata standards are widely used to describe especially tabular data in depth, including their columns/variables. According to re3data.org, a global registry of research data repositories, they are the most prevalent standards with these capabilities. Additionally, re3data.org identifies OAI-PMH as the most widely adopted protocol for harvesting such metadata, with many endpoints available at re3data.org. Building on this, Wenzig and Han (2024, doi:10.29173/iq1116) recently analysed over 250,000 data records in the DDICodebook format from various sources. Their methods can be adapted for continuous monitoring of metadata usage and availability. Such an approach could help new users identify where and how other institutions implement DDI standards, offering concrete starting points for further research and community engagement. A project funded by KonsortSWD –NFDI4Society is advancing these efforts with the following key deliverables: •Development of metrics on DDI standard adoption, including the number of institutions using DDI, number of institutions providing DDI metadata through standardized methods, and statistics about the availability of records in the different DDI standards (DDI-Codebook, DDI-Lifecycle, DDI-CDI). •Publication of datasets containing raw data for these metrics from each survey crawl. •An experimental dashboard displaying these metrics alongside links to individual metadata records and repositories. This talk will provide an inside on the project’s results, which serves two immediate goals: 1. Visualizing the adoption of DDI metadata standards. 2. Encouraging institutions that have not yet made their metadata accessible to do so, enhancing their visibility within the community. Metadata Challenges DDI Adoption Metrics • Knut Wenzig • EDDI2025 Budapest • 3 December 2025 3 •Metadata are needed: There are numerous compelling use cases demonstrating that making metadata publicly available at a fine-grained level –such as for variables and their categories in tabular data –would deliver significant value: think about an LLM driven chatbot for data. But this metadata should be standardised and published in standardised ways. •Evidence is needed –Example from a review: “Further, there is no reason given why DDI was chosen here. … However, at this point it seems arbitrary. Why not look at any of the 500+ other formats?” •FAIR principles F2/A1/I1/R1 want to be met: Rich metadata in a broadly accepted language via standardised ways. What Could Drive Metadata Publication? –Perhaps Some Metrics DDI Adoption Metrics • Knut Wenzig • EDDI2025 Budapest • 3 December 2025 4 Where can we find and access them? •re3data.org is a global registry of 4,328 research data repositories (Nov 2025) o52 metadata standards / 9 different APIs •Alternative options: oExplore.OpenAIRE.eu reports >11,000 repositories with less detailed information oFAIRsharing.org (1,730 repositories) could be an additional source oDDI Agency Registry (139 agencies, Nov 2025) but API needs at least documentation Usage of Top10 Metadata Standards Reported by re3data.org DDI Adoption Metrics • Knut Wenzig • EDDI2025 Budapest • 3 December 2025 5 695 522 324 245 221 200 101 71 69 62 0 100 200 300 400 500 600 700 800 Dublin Core DataCite Metadata Schema DDI Data Documentation Initiative Repository -Developed Metadata Schemas ISO 19115 OAI-ORE - Open Archives Initiative Object… FGDC/CSDGM - Federal Geographic Data… schema.org JSON-LD OpenAIRE Guidelines Data as of November 2025, source: https://www.re3data.org/search •Usage of other standards with similar features (information on variable level) increases over time –schema.org is new in the list. The Most Relevant Members of the DDI Standards Family DDI Adoption Metrics • Knut Wenzig • EDDI2025 Budapest • 3 December 2025 6 The DDI standards family, rooted in social survey research, can describe not only survey data but also generic tabular data (with objects in rows and their properties as variables in columns). •DDI-Codebook: Describes datasets, focusing particularly on their variables, including categories, concepts, and descriptive statistics. •DDI-Lifecycle: Adds features for describing entire questionnaires and linking questions to variables. •DDI-CDI (Cross Domain Integration): Focuses on data in a more domain-neutral way and can describe data management processes and provenance. Use of APIs Reported by re3data.org DDI Adoption Metrics • Knut Wenzig • EDDI2025 Budapest • 3 December 2025 7 818 560 288 274 126 80 47 45 44 0 100 200 300 400 500 600 700 800 900 REST OAI-PMH other FTP SWORD NetCDF OpenDAP SOAP SPARQL Data as of November 2025, source: https://www.re3data.org/search •REST requires customized access software for each repository. •API responses show a lack of standardized answers. •OAI-PMH is a protocol developed for harvesting XML metadata. Wenzig and Han (2024, doi:10.29173/iq1116) analyzed over 250,000 data records in DDI-Codebook format. Learnings from 1st Crawl in March 2025 DDI Adoption Metrics • Knut Wenzig • EDDI2025 Budapest • 3 December 2025 8 •268 repositories were identified as using DDI and not marked as closed on re3data.org. •Evaluation of OAI-PMH Endpoint URLs (retrieved via re3data.org REST API): o108 repositories needed an endpoint URL update. oA new URL was suggested for 43 repositories (mainly Dataverse). oEndpoint URL deletion was suggested for 6 repositories. o1 repository was closed. o31 repositories had up-to-date endpoint URLs, no update needed. o79 repositories don’t support OAI-PMH, no update needed. •Learning: Data quality is poor, so curation of information from re3data.org will remain necessary in the medium term. Learnings from 2nd Crawl in July 2025 DDI Adoption Metrics • Knut Wenzig • EDDI2025 Budapest • 3 December 2025 9 •289 repositories were identified as using DDI and not marked as closed on re3data.org, 198 report to use OAI-PMH, re3data.org updated API. •Comparison with 1st round: o10 Dataverses with wrong endpoints o1 reported to not use DDI anymore, 4 updated name o22 repositories were new o1 repo suggested to be added (could be out of scope?) o10 repos were offline, data collected manually •Learning: Constant flux. Some endpoints down because of crawlers. Feedback for Colectica improved OAI-PMH implementation. What could be additional sources? 16 DDI Adoption: Show strength of Ecosystem and Guide New Users Elements of a dashboard •Display each known publisher (with source of information) •Display some basic adoption metrics •Display basic information (repo name, software, standard, object count, link to sample record) •https://ddi-repo-viz.lovable.app/ Final remarks DDI Adoption Metrics • Knut Wenzig • EDDI2025 Budapest • 3 December 2025 17 •The latest data are included with these slides: https://doi.org/10.5281/zenodo.17760658 (linked to previous versions) •If you are interested in these topics, consider joining the Metadata Publication and Access Working Group. •Your recommendations for suitable journals would be greatly appreciated. Thanks for your attention. DIW Berlin — German Institute for Economic Research Mohrenstraße 58, 10117 Berlin www.diw.de/en Redaktion www.diw.de/100jahre Knut Wenzig –[email protected] –https://orcid.org/0000-0002-2259-0203 Socio-Economic Panel (SOEP) License DDI Adoption Metrics • Knut Wenzig • EDDI2025 Budapest • 3 December 2025 19 Unless otherwise stated this presentation is offered under license CC BY-SA 4.0. The license does not apply to the following copyrighted material used in this presentation: the logo and artwork of DIW Berlin