Publishing Fine-Grained Standardized Metadata – Lessons Learned from Three Research Data Centers

Working Paper 2025 12 Publishing Fine-Grained Standardized Metadata – Lessons Learned from Three Research Data Centers Knut Wenzig, Andreas Daniel, Dominique Hansen, Tobias Koberg, Mihaela Tudose Mai 2025 www.konsortswd.de Seite 1 von 16 Publishing Fine-Grained Standardized Metadata – Lessons Learned from Three Research Data Centers Knut Wenzig, [email protected], https://orcid.org/0000-0002-2259-0203, SOEP, DIW Berlin, Germany Andreas Daniel, dani[email protected], https://orcid.org/0000-0002-0111-8858, DZHW, Hannover, Germany Dominique Hansen, [email protected], SOEP, DIW Berlin, Germany Tobias Koberg, tobias.ko[email protected], LIfBi, Bamberg, Germany Mihaela Tudose, [email protected], LIfBi, Bamberg, Germany Berlin, Mai 2025 doi:10.5281/zenodo.15464650 Abstract FAIRness of research data largely depends on the availability of rich and standardized metadata. While such metadata is commonly available at the study level, fine-grained metadata, especially for tabular scientific data, is often lacking (Wenzig/Han 2024). In a project funded by KonsortSWD-NFDI4Society, three research data centers (SOEP, LIfBi, and DZHW) investigate what is required to convert existing metadata into a standardized format and publish it using a common protocol. Seite 2 von 16 1. Motivation Fine-grained standardized metadata for tabular research data is rarely accessible. The global registry of research data repositories – re3data.org – lists more than 3,300 repositories that collectively provide over 250,000 metadata resources in DDI-Codebook format. However, only one repository uses the OAI-PMH protocol and only 0.7% of the records contain metadata on the fine-grained variable level (Wenzig and Han 2024). The problem is not the lack of availability of standards: Metadata standards from the DDI family (DDI-Codebook and DDI-Lifecycle, and DDI-CDI—DDI Alliance 2012, 2020, 2025) document research data at a fine granular level, including information about data sets, variables, and their characteristics. Until 2025, these DDI standards only specified metadata in the form of XML files. (This changed with the model driven approach of DDICDI and will continue with the next version of DDI-Lifecycle.) OAI-PMH (Open Archives Initiative 2015) provides an established protocol that enables metadata sharing and low-level access. It is built for XML metadata and allows for harvesting metadata from a repository by the URL of the OAI-PMH endpoint. Why is the publication of fine-grained standardized metadata important? While many research data centers already provide high-quality data and metadata services, there are compelling reasons to consider more standardized approaches: ▪ In principle, standardization can improve the scalability of business processes and free up resources. ▪ Standardized metadata can be re-used within the Open Data Format (Han, Hartl, and Wenzig 2024) to reduce the cost of managing data files for different software platforms and to deliver multilingual documentation directly to the user’s statistical software. ▪ AI, especially LLMs, can make use of fine-grained standardized metadata in unknown ways – if those metadata are available. The publication of fine-grained standardized metadata would also contribute to the creation of FAIR research data and facilitate the FAIR-Principles (Wilkinson et al. 2016): ▪ They increase findability, because they provide more content and deeper insights for each kind of search. ▪ Since datasets in the social sciences are, in most cases, not freely accessible, it is even more important that relevant metadata on the variables - the core units of analysis - are openly available. Furthermore, if the data is no longer accessible at all, there are good reasons (e.g. more open licenses on metadata facilitate Seite 3 von 16 harvesting and storage by third parties) to expect that metadata are still available and, with them, valuable information, especially on the variable level. ▪ Providing the metadata at the variable level in a formal language would contribute to the interoperability of the data, by improving its comprehensibility for both humans and machines. Using an established standard, like DDI, for the metadata would further enhance interoperability by ensuring consistency and compatibility across different systems. ▪ Distributing information in domain-relevant standards will result in a betterinformed re-use, as field definitions and general descriptions remain comprehensible over time. The provision of fine-grained standardized metadata offers many advantages, as it enhances the machine actionability and the overall FAIRness of the data. Therefore, it should be considered best practice. However, as outlined, standardized metadata at the variable level remains scarce. This raises the question of whether factors beyond the availability of standards and accessways pose significant obstacles to data providers in generating such metadata. As we demonstrate, this does not appear to be the case, as the necessary metadata is already available and can be published cost-effectively. Stata and SPSS are widely used in economics and social sciences. Their file formats already contain metadata on variable level, including variable names, variable labels, and categories/value labels. The main challenge is to map the metadata to an existing standard, integrating the mapping the data processing workflows, and making the metadata accessible. In the following we outline our approach by addressing these challenges based on the metadata of three data centers. 2. Three Starting Points All three partners already manage metadata on the variable level. These are presented to end-users via different portals: www.paneldata.org, variablesearch.lifbi.de, and metadata.fdz.dzhw.eu. The data base for these portals is as heterogenous as the metadata management philosophies: ▪ SOEP manages its metadata in CSV tables on a GitLab server with standard office software (Wenzig 2019). ▪ At LIfBi, an MS SQL database is the backbone for the metadata, which are managed using a self-developed metadata editor (Wenzig et al. 2016). ▪ DZHW uses a NoSQL database (MongoDB) to store metadata in JSON files. Metadata ingestion is facilitated through metadata editor with input masks for Seite 4 von 16 general metadata fields, while variable and question metadata are imported using a dedicated JSON importer. Prior to this project, the partners used this metadata for documentation on customdeveloped web portals, but did not officially publish metadata on variable level in a standardized manner. During the first project phase, the partners committed to target a payload on each of the relevant conceptual levels because this information is already (somehow) available in the three RDCs: ▪ On the study level, the information that is already used for registering the datasets’ DOIs could be re-used. This means that, as a first approach, the digital object for which the DOI is registered should be the one to publish metadata in DDICodebook. ▪ On the dataset level, the file names (with or without suffix), a label, and a description should be recorded. ▪ On variable level, the name of the variable, its label and an optional description should be available. Pairs of value and labels build the categories (the term used in DDI-Codebook) or value labels (Stata or SPSS terms). It should also be possible to include question texts and all kinds of key words or overarching concepts. ▪ Each metadata field containing natural language should be assigned a language code. It should be possible to provide alternative translations for those entries. 3. DDI-Codebook An XML file in DDI-Codebook standard (DDI Alliance 2012) can have five elements below the root-element <codeBook>: <docDscr>, <stdyDscr>, <fileDscr>, <dataDscr>, and <otherMat>. ▪ <docDscr> contains bibliographic information describing the XML file itself. This is not addressed in the following as it should only contain information about the metadata itself, i.e. who created it, creation date etc. ▪ The Study Description in <stdyDscr> consists of information about the data collection, study, or compilation that this DDI-compliant documentation file describes. ▪ Within <fileDscr>, information about the data file(s) that comprises a collection can be stored. ▪ The section <dataDscr> contains the descriptions of the variables, the fine-grained metadata that are in the center of this project. Seite 5 von 16 ▪ Finally, the element <otherMat> allows for the inclusion of other materials that are related to the study. The tree structure (DDI Alliance n.d.) provides a good overview of the DDI-Codebook standard, the field-level documentation (DDI Alliance 2014) serves as the reference book for each element. The use case we show here is an idealized example derived from the actual holdings of the three RDCs. Figure 1 gives an impression of the material we want to document in a DDI-Codebook XML file: Within a study we have two tables (in Stata format) tab1.dta and tab2.dta that are packed to a ZIP file. Each of the two tables contains two variables. Figure 1: Use Case with two Stata-Files, each containing two Variables, in one ZIP-File within a Study. tab1varA tab1varB tab2varX tab2varY tab1.dta tab2.dta data.zip TheStudy Figure 2: Content of element <stdyDscr> <stdyDscr> <citation> <titlStmt> <titl xml:lang="de">TheStudyTitleGerman</titl> <parTitl xml:lang="en">TheStudyTitleEnglish</parTitl> <IDNo agency="DOI">studyDOI</IDNo> </titlStmt> </citation> </stdyDscr> As the study level is of minor importance for this project, the XML code for the element <stdyDscr> in Figure 2 only shows also the single mandatory field of the DDI-Codebook standard <titl> (which implies <citation>, <titlStmt>, and <stdyDscr> are also present). <parTitl> shall be used to provide a translated title. Element <IDno> contains the registered DOI. So, in our scenario the <stdyDscr> (Figure 1) should contain information Seite 6 von 16 on the DOI level, which means for the three RDCs that metadata used for the DOI registration (like study title, author information or the scope of the study) can be re-used. 1 The attribute xml:lang is used to specify the language of the field’s content. The ISO 2letter codes, like “de” for German or “en” for English, should be used. Figure 3: Content of element <fileDscr> <fileDscr ID="Filename"> <fileTxt> <fileName xml:lang="de">FileLabelGerman</fileName> <fileName xml:lang="en">FileLabelEnglish</fileName> <fileCont xml:lang="de">FileDescriptionGerman</fileCont> <fileCont xml:lang="en">FileDescriptionEnglish</fileCont> </fileTxt> </fileDscr> The element <fileDscr> (Figure 3) should contain information on file level and can be repeated for collections with multiple files. The attribute ID can be used for the filename; in our use case it could be data.zip. In the next section of the DDI-Codebook XML file, this ID can be used to assign a variable to a file. If a repository provides files in different formats, the element should be repeated to describe the two separate files, it also could make sense to use the file name without extension. Within the element <fileTxt>, the elements <fileName> and <fileCont> contain a label and a description of for the dataset. Again, the attribute “xml:lang” is used to indicate the language of the descriptive text. It is not always possible to fully describe the structure of a collection containing multiple files across different folders or bundled in a ZIP file. However, if ZIP files (or similar archives) are used, one option is to name all files explicitly—using the <fileCont> element for both the ZIP file and its contents to indicate their relationship, at least in a humanreadable way. 1 The three project partners register DOIs through da|ra, which provides metadata via OAI-PMH (https://www.da-ra.de/oaip/), including DDI-Lifecycle format based on the mapping by Koch et al. (2017, pp. 91-92). The da|ra service will be discontinued by the end of 2025, but as a DataCite agent, its metadata records will remain accessible through DataCite (https://datacite.org/). Seite 7 von 16 Figure 4: Content of element <dataDscr> <dataDscr> <varGrp type="file" var="var1ID var2ID …" name="tab1.dta"> <labl xml:lang="en">tab1.dta-LabelEnglish</labl> <labl xml:lang="de"> tab1.dta-LabelGerman</labl> <txt xml:lang="en"> tab1.dta-DescriptionEnglish</txt> <txt xml:lang="de"> tab1.dta-DescriptionGerman</txt> </varGrp> <varGrp type="file" var="var3ID var4ID …" name="tab2.dta"> … </varGrp> <var ID="var1ID" name="tab1varA" files=xs:IDREFS> <location fileid="FileName"/> ... </var> ... <var ID="var4ID" name="tab2varY"> <location fileid="FileName"/> ... </var> </dataDscr> The specification of DDI-Codebook does not locate the information on variables within the element <fileDscr> but introduces the element <dataDscr> (Figure 4) that holds information on each of the variables. The element <dataDscr> is repeatable and if one would want to use the attribute ID for the file name, one could organize the variables of multiple files. The element <varGrp> can be used to group the variables in datasets. Therefore, the attribute “type” must be “file”, the attribute “name” can be used to provide the name of the dataset and the elements <labl> and <txt> can be used to store a label and a description of the dataset. Multilingual content can be accommodated using the attribute xml:lang. Each variable has its own element <var> within <dataDscr>. The name of the variable is encoded in the attribute “name”. The element <location> with its attribute “fileid” is used to assign a variable to a file/dataset. This “fileid” should relate to the content of attribute “ID” in element <fileDscr>. There can also the attribute files in <var> but it is not clear when this attribute should be used and when the element <location> should be used. Seite 8 von 16 Overall, there appear to be too many ways to organize the variables of multiple data tables in DDI-Codebook: While the element <varGrp> may be the preferred way, but one could expect that some people will use some repetition of the element of <dataDcsr> to group variables, others use the element <location> to link variables to files, and others deploy the attribute “files” in element <var>. From the perspective of potential harvesters this variety of options seems not to be optimal. The element <var> also is the container for more information on the variables. (Figure 5) Figure 5: Content of element <var>. <var name="VariableName"> <location fileid="..."/> <labl xml:lang="en">VariableLabelEnglish</labl> <labl xml:lang="de">VariableLabelGerman</labl> <qstn xml:lang="en">QuestionTextEnglish</qstn> <qstn xml:lang="de">QuestionTextGerman</qstn> <txt xml:lang="en">VariableDescriptionEnglish</txt> <txt xml:lang="de">VariableDescriptionGerman</txt> <catgry> <catValu>Value</catValu> <labl xml:lang="de">ValueLabelGerman</labl> <labl xml:lang="en">ValueLabelEnglish</labl> </catgry> <catgry>...</catgry> <concept xml:lang="en">ConceptLabelEnglish</concept> <concept xml:lang="de">ConceptLabelGerman</concept> </var> The element <labl> holds the variable label as known from statistical software like Stata or SPSS. If available, the element <qstn> can be used to store the precise wording of a question. Any other descriptive texts can go into the element <txt>. The element <catgry> holds a pair of the elements <catValu> and <labl>, the latter with optional alternatives in different languages. It must be repeated for each labelled value of the variable. Finally, the repeatable element <concept> can contain information like topics, terms from thesauri, or other keyword systems. Again, alternatives can be provided by usage of the attribute xml:lang. The three project partners exported the information from their metadata systems. Alternatively, most of the information could have been derived from the Stata or SPSS dataset files: file names, variable names, variable labels, and value labels. It is easy to extract them and convert this information to a DDI-Codebook file, for example by using Seite 15 von 16 Van de Sompel, H. et al. (2023) FAIR Signposting Profile. Available at: https://signposting.org/FAIR/ (Accessed: 18 February 2025). Wenzig, K. (2019) Longitudinal Metadata at the Socio-Economic Panel. Available at: http://doi.org/10.5281/zenodo.3554859 (Accessed: 18 February 2025). Wenzig, K. and Han, X. (2024) ‘State of DDI Cloud’, IASSIST Quarterly, 48(4). Available at: http://doi.org/10.29173/iq1116 (Accessed: 18 February 2025). Wenzig, K., et al. (2016) ‘Management of Metadata: An Integrated Approach to Structured Documentation’ in: H.-P. Blossfeld et al. (eds.) Methodological Issues of Longitudinal Surveys. Wiesbaden: Springer VS, pp. 627-647. Available at: https://doi.org/10.1007/9783-658-11994-2_35 Wilkinson, M. et al. (2016) ‘The FAIR Guiding Principles for scientific data management and stewardship’. Scientific Data 3, 160018. Available at: https://doi.org/10.1038/sdata.2016.18 (Accessed: 18 February 2025). Seite 16 von 16 Impressum Kontakt DIW Berlin Sozio-ökonomisches Panel Knut Wenzig https://www.diw.de/staff/de/kwenzig [email protected] Berlin, Mai 2025 KonsortSWD Working Paper KonsortSWD baut als Teil der Nationalen Forschungsdateninfrastruktur Angebote zur Unterstützung von Forschung mit Daten in den Sozial-, Verhaltens-, Bildungsund Wirtschaftswissenschaften aus. Unsere Mission ist es, die Forschungsdaten-infrastruktur zur Beforschung der Gesellschaft zu stärken, zu erweitern und zu vertiefen. Sie soll nutzungsorientiert ausgestaltet sein und die Bedürfnisse der Forschungscommunities berücksichtigen. Wichtiger Grundstein ist dabei das seit über zwei Jahrzehnten durch den Rat für Sozialund Wirtschaftsdaten (RatSWD) aufgebaute Netzwerk von Forschungsdatenzentren. In dieser Reihe erscheinen Beiträge rund um das Forschungsdatenmanagement, die im Kontext von KonsortSWD entstehen. Beiträge, die extern und doppelblind begutachtet wurden sind entsprechend gekennzeichnet. KonsortSWD wird im Rahmen der NFDI durch die Deutsche Forschungsgemeinschaft (DFG) gefördert – Projektnummer: 442494171. Diese Veröffentlichung ist unter der Creative-Commons-Lizenz (CC BY 4.0) lizenziert. DOI: 10.5281/zenodo.15464650 Zitationsvorschlag Wenzig, K., Daniel, A., Hansen, D., Kober, T. & Tudose, M. (2025). Publishing Fine-Grained Standardized Metadata – Lessons Learned from Three Research Data Centers. KonsortSWD Working Paper 12/2025. Konsortium für die Sozial-, Verhaltens-, Bildungsund Wirtschaftswissenschaften (KonsortSWD). doi:10.5281/zenodo.15464650