Open Data Format with Stata, R & Python: Multiplatform Distribution Powered by DDI-Codebook at Low Cost

Open Data Format with Stata, R & Python Multiplatform Distribution powered by DDI-Codebook at Low Cost Knut Wenzig and Xiaoyao Han EDDI 2025, Budapest, 1 December 2025 doi:10.5281/zenodo.17770570, License: CC BY-SA 4.0 (exceptions see last slide) Motivation Open Data Format • Knut Wenzig and Xiaoyao Han • EDDI2025 Budapest • 1 December 2025 2 •Typical use case: •Metadata at variable level are often managed in workflows using statistical software. •After data curation data files were transformed in different formats. •Results: •Poor re-use of those metadata. •Metadata don’t drive the process. •Why not adopt a metadata-centric data distribution that is platformindependent and includes import routines for various statistical software? •Additionally, it is highly suitable for long-term preservation. •Increases FAIRness of data products One Format for Many Users Open Data Format • Knut Wenzig and Xiaoyao Han • EDDI2025 Budapest • 1 December 2025 3 ODF … Stata Python R … Open Data Format in a Nutshell Open Data Format • Knut Wenzig and Xiaoyao Han • EDDI2025 Budapest • 1 December 2025 4 Data in CSV Format Open Data Format • Knut Wenzig and Xiaoyao Han • EDDI2025 Budapest • 1 December 2025 5 •UTF-8 encoding •Comma separated •Text in quotes •Variable names in first line bap87,bap9201,bap9001,bap9002,bap9003,bap96,name 4,-2,1,-1,2,-2,"Jakob" 3,5,-2,1,4,1.57,"Luca" ,-1,-1,2,-1,1.92,"Emilia" 1,9,-2,2,4,1.85,"Charlotte" -1,4,2,3,1,1.91,"Johanna" 3,4,-1,4,-2,1.8,"Paul" 1,9,2,-1,-1,1.8,"" •At the moment we do not enforce or validate this. •Our software packages use the standard import routines of different platforms. •There can be similar problems known from CSV: encoding, column separation, variable names, precision Metadata is XML file in DDI-Codebook Open Data Format • Knut Wenzig and Xiaoyao Han • EDDI2025 Budapest • 1 December 2025 6 •Metadata will be valid DDI-Codebook version 2.6. •Contains only the information, which is necessary for rich use across different software platforms and across disciplines •Multilingual with options for deep links to metadata portals. •Profile for CESSDA’s metadata validator available. •(JSON at the moment only contains version information) <dataDscr> <var name="bap87"> <labl xml:lang="en">Current Health</labl> <labl xml:lang="de">Gesundheitszustand gegenwärtig</labl> <txt xml:lang="en">Question: How would you describe your current health?</txt> <txt xml:lang="de">Frage: Wie würden Sie Ihren gegenwärtigen Gesundheitszustand ...</txt> <notes> <ExtLink URI="https://paneldata.org/soep-core/data/bap/bap87"/> </notes> … Metadata on Study Level Open Data Format • Knut Wenzig and Xiaoyao Han • EDDI2025 Budapest • 1 December 2025 7 •<titl> is the only mandatory element in DDI-Codebook <stdyDscr> <citation> <titlStmt> <titl>DatasetStudy</titl> </titlStmt> </citation> </stdyDscr> Metadata on Dataset Level Open Data Format • Knut Wenzig and Xiaoyao Han • EDDI2025 Budapest • 1 December 2025 8 <fileDscr> <fileTxt> <fileName>DatasetName</fileName> <fileCitation> <titlStmt> <titl xml:lang="en">DatasetLabel (English)</titl> <parTitl xml:lang="de">DatasetLabel (German)</parTitl> </titlStmt> </fileCitation> <fileCont xml:lang="en">DatasetDescription (English)</fileCont> <fileCont xml:lang="de">DatasetDescription (German)</fileCont> </fileTxt> <notes> <ExtLink URI="DatasetURL"/> </notes> </fileDscr> •Mulitlingual components omitted in next slides. Metadata on Variable Level Open Data Format • Knut Wenzig and Xiaoyao Han • EDDI2025 Budapest • 1 December 2025 9 <dataDscr> <var name="VariableName"> <labl xml:lang="en">VariableLabel (English)</labl> <txt xml:lang="en">VariableDescription (English)</txt> <catgry> <catValu>VariableValue (e.g. -1, 1)</catValu> <labl xml:lang="en">VariableValueLabel (English)</labl> </catgry> <catgry> [...] </catgry> <varFormat type="VariableType (`numeric` or `character`)"/> <notes> <ExtLink URI="VariableURL"/> </notes> </var> <var name=[...]> [...] </var> </dataDscr> License Open Data Format • Knut Wenzig and Xiaoyao Han • EDDI2025 Budapest • 1 December 2025 16 Unless otherwise stated this presentation is offered under license CC BY-SA 4.0. The license does not apply to the following copyrighted material used in this presentation: the logo and artwork of DIW Berlin