Open Science Best Practices for Source Code Preservation

Open Science Best Practices for Source Code Preservation Stefano Zacchiroli Software Heritage Télécom Paris, Polytechnic Institute of Paris 30 March 2023 CERN Geneva, Switzerland THE GREAT LIBRARY OF SOURCE CO DE Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 1 / 32 Outline 1Introduction 2The software commons: an endangered knowledge 3Avoiding the tragedy of an unattended software commons 4Source code in open science 5Archive and reference 6Describe, cite, credit 7Policy framework 8Call to actions Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 2 / 32 About the speaker Professor of Computer Science, Télécom Paris, Institut Polytechnique de Paris Research: digital commons, open source software engineereing, computer security, software supply chain. 70+ papers in these fields: upsilon.cc/zack/research/publications Free/Open Source Software activist (20+ years) Debian Developer & Former 3x Debian Project Leader Former Open Source Initiative (OSI) director Software Heritage co-founder & CTO Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 2 / 32 Software is everywhere (and a key mediator) in society Software Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 3 / 32 Free/Open Source Software (FOSS) is everywhere as well Definition (Free Software (1986)) A program is free software if the program’s users have the four essential freedoms: Freedom #0, to run the program, for any purpose Freedom #1, to study how the program works, and change it Freedom #2, to redistribute copies Freedom #3, to improve the program, and release improvements Current estimates: 99% of software products on the market contains at least some free software parts (Synopsis 2020). Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 4 / 32 The Commons and FOSS Definition (Commons) The commons is the cultural and natural resources accessible to all members of a society, including natural materials such as air, water, and a habitable earth. These resources are held in common, not owned privately. Definition (Software Commons) The software commons consists of all computer software which is available at little or no cost and which can be altered and reused with few restrictions. Thus all open source software and all free software are part of the [software] commons. [...] Kranich and Schement (2008); Schweik and English (2012). Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 5 / 32 Software is dual-form knowledge “The source code for a work means the preferred form of the work for making modifications to it." GPL Licence Hello World Program (excerpt of binary) 4004e6: 55 4004e7: 48 89 e5 4004ea: bf 84 05 40 00 4004ef: b8 00 00 00 00 4004f4: e8 c7 fe ff ff 4004f9: 90 4004fa: 5d 4004fb: c3 Program (source code) /* Hello World program */ #include<stdio.h> void main() { printf("Hello World"); } Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 6 / 32 Software is dual-form knowledge “The source code for a work means the preferred form of the work for making modifications to it." GPL Licence Hello World Program (excerpt of binary) 4004e6: 55 4004e7: 48 89 e5 4004ea: bf 84 05 40 00 4004ef: b8 00 00 00 00 4004f4: e8 c7 fe ff ff 4004f9: 90 4004fa: 5d 4004fb: c3 Program (source code) /* Hello World program */ #include<stdio.h> void main() { printf("Hello World"); } Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 6 / 32 Software is dual-form knowledge “The source code for a work means the preferred form of the work for making modifications to it." GPL Licence Hello World Program (excerpt of binary) 4004e6: 55 4004e7: 48 89 e5 4004ea: bf 84 05 40 00 4004ef: b8 00 00 00 00 4004f4: e8 c7 fe ff ff 4004f9: 90 4004fa: 5d 4004fb: c3 Program (source code) /* Hello World program */ #include<stdio.h> void main() { printf("Hello World"); } Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 6 / 32 But where is this commons? many disparate development platforms, with a few dominant players (e.g., GitHub) a myriad places where distribution may happen most of them operated by for-profit companies Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 8 / 32 Software source code is fragile Like all digital information, FOSS is fragile link rot: projects are created, moved around, removed business-driven code loss (e.g., Gitorious, Google Code, Bitbucket) data rot: physical media with legacy software decay If a website disappears you go to the Internet Archive... where do you go if (a repository on) GitHub or GitLab goes away? Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 9 / 32 Software source code is fragile Like all digital information, FOSS is fragile link rot: projects are created, moved around, removed business-driven code loss (e.g., Gitorious, Google Code, Bitbucket) data rot: physical media with legacy software decay If a website disappears you go to the Internet Archive... where do you go if (a repository on) GitHub or GitLab goes away? Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 9 / 32 Calling for preservation: UNESCO UNESCO, Inria, Software Heritage invite 40 international experts meet in Paris ... The call is published on Feb 2019 “[We call to] support efforts to gather and preserve the artifacts and narratives of the history of computing, while the earlier creators are still alive” https://en.unesco.org/foss/paris-call-software-source-code Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 10 / 32 Calling for preservation: UNESCO UNESCO, Inria, Software Heritage invite 40 international experts meet in Paris ... The call is published on Feb 2019 “[We call to] support efforts to gather and preserve the artifacts and narratives of the history of computing, while the earlier creators are still alive” https://en.unesco.org/foss/paris-call-software-source-code Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 10 / 32 Calling for preservation: UNESCO UNESCO, Inria, Software Heritage invite 40 international experts meet in Paris ... The call is published on Feb 2019 “[We call to] support efforts to gather and preserve the artifacts and narratives of the history of computing, while the earlier creators are still alive” https://en.unesco.org/foss/paris-call-software-source-code Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 10 / 32 Calling for preservation: Donald Knuth and Len Shustek Communications of the ACM, February 2021 "Telling historical stories is the best way to teach. It’s much easier to understand something if you know the threads it is connected to." Let’s Not Dumb Down the History of Computer Science Donald E. Knuth, Len Shustek https://doi.org/10.1145/3442377 A unique opportunity most of the creators are still here: we can talk to them! but the clock is ticking... Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 11 / 32 Calling for preservation: Donald Knuth and Len Shustek Communications of the ACM, February 2021 "Telling historical stories is the best way to teach. It’s much easier to understand something if you know the threads it is connected to." Let’s Not Dumb Down the History of Computer Science Donald E. Knuth, Len Shustek https://doi.org/10.1145/3442377 A unique opportunity most of the creators are still here: we can talk to them! but the clock is ticking... Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 11 / 32 Why Open Science? Open Science (Second National Plan for Open Science, France, 2021) Unhindered dissemination of results, methods and products from scientific research. It draws on the opportunity provided by recent digital progress to develop open access to publications and – as much as possible – data,source code and research methods. Jean-Eric Paquet (EU DGRI, on the objective of Open Science) “Increase scientific quality, the pace of discovery and technological development, as well as societal trust in science.” Mariya Gabriel (EU Commissionner for Research) The COVID-19 crisis has also shown that cooperation at international level in research and innovation is more important than ever, including through open access to data and results.No nation, no country can tackle any of these global challenges alone. Yuval Noah Harari (on COVID 19) “The real antidote [to epidemic] is scientific knowledge and global cooperation.” Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 12 / 32 Why Open Science? Open Science (Second National Plan for Open Science, France, 2021) Unhindered dissemination of results, methods and products from scientific research. It draws on the opportunity provided by recent digital progress to develop open access to publications and – as much as possible – data,source code and research methods. Jean-Eric Paquet (EU DGRI, on the objective of Open Science) “Increase scientific quality, the pace of discovery and technological development, as well as societal trust in science.” Mariya Gabriel (EU Commissionner for Research) The COVID-19 crisis has also shown that cooperation at international level in research and innovation is more important than ever, including through open access to data and results.No nation, no country can tackle any of these global challenges alone. Yuval Noah Harari (on COVID 19) “The real antidote [to epidemic] is scientific knowledge and global cooperation.” Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 12 / 32 Software is a pillar of Open Science Software powers modern research [...] software [...] essential in their fields. Top 100 papers (Nature, 2014) Sometimes, if you dont have the software, you dont have the data Christine Borgman, Paris, 2018 A key pillar: software (source code) The links in the picture are important Nota Bene software may be a tool, a research outcome and a research object access to the source code is essential! Preserving the history of source code is important for reproducibility Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 13 / 32 Software is a pillar of Open Science Software powers modern research [...] software [...] essential in their fields. Top 100 papers (Nature, 2014) Sometimes, if you dont have the software, you dont have the data Christine Borgman, Paris, 2018 A key pillar: software (source code) The links in the picture are important Nota Bene software may be a tool, a research outcome and a research object access to the source code is essential! Preserving the history of source code is important for reproducibility Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 13 / 32 Outline 1Introduction 2The software commons: an endangered knowledge 3Avoiding the tragedy of an unattended software commons 4Source code in open science 5Archive and reference 6Describe, cite, credit 7Policy framework 8Call to actions Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 14 / 32 Software Heritage in a nutshell www.softwareheritage.org THE GR EAT L IB RARY OF S OURC E CO DE Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve and share all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 14 / 32 Software Heritage in a nutshell www.softwareheritage.org THE GR EAT L IB RARY OF S OURC E CO DE Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve and share all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 14 / 32 Software Heritage in a nutshell www.softwareheritage.org THE GR EAT L IB RARY OF S OURC E CO DE Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve and share all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 14 / 32 Software Heritage in a nutshell www.softwareheritage.org THE GR EAT L IB RARY OF S OURC E CO DE Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve and share all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 14 / 32 The largest public source code archive, principled bit.ly/swhpaper archive.softwareheritage.org Technology transparency and FOSS replicas all the way down Content (billions!) intrinsic identifiers facts and provenance Organization non-profit multi-stakeholder Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 15 / 32 The largest public source code archive, principled bit.ly/swhpaper archive.softwareheritage.org Technology transparency and FOSS replicas all the way down Content (billions!) intrinsic identifiers facts and provenance Organization non-profit multi-stakeholder Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 15 / 32 A peek under the hood: a global view on the software commons Git loader Mercurial loader Debian source package loader pypi source package loader . . . Software Heritage Archive Merkle DAG + blob storage Loading & deduplication dsc dsc hg hg hg git git git git svn svn svn tar zip software origins Package repos Forges GitHub lister GitLab lister Debian lister PyPi lister . . . Distros ... Scheduling Listing (full/incremental) tar origins snapshots releases revisionsrevisions directoriesdirectories contents Aglobal graph linking together fully deduplicated source code artifact (files, commits, directories, releases, etc.) to the places that distribute them (e.g., Git repositories), providing a unified view on the entire Software Commons. (Size: ~30 B nodes, ~300 B edges, ~1 PiB blobs) Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 16 / 32 Outline 1Introduction 2The software commons: an endangered knowledge 3Avoiding the tragedy of an unattended software commons 4Source code in open science 5Archive and reference 6Describe, cite, credit 7Policy framework 8Call to actions Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 18 / 32 Source code in Open Science: a plurality of needs to address Researchers archive and reference software used in articles find useful software get credit for developed software verify, reproduce, improve results Laboratories/teams track software contributions produce reports maintain web page Research Organizations and/or Funders know its software assets technology transfer impact metrics funding strategy career evaluation Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 18 / 32 Source code in Open Science: a plurality of needs to address Researchers archive and reference software used in articles find useful software get credit for developed software verify, reproduce, improve results Laboratories/teams track software contributions produce reports maintain web page Research Organizations and/or Funders know its software assets technology transfer impact metrics funding strategy career evaluation Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 18 / 32 Source code in Open Science: a plurality of needs to address Researchers archive and reference software used in articles find useful software get credit for developed software verify, reproduce, improve results Laboratories/teams track software contributions produce reports maintain web page Research Organizations and/or Funders know its software assets technology transfer impact metrics funding strategy career evaluation Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 18 / 32 Archive, Reference, Describe, Cite and Credit Archive Research software artifacts must be properly archived make sure we can retrieve them (reproducibility) Reference Research software artifacts must be properly referenced make sure we can identify them (reproducibility) Describe Research software artifacts must be properly described make it easy to discover and reuse them (visibility) Cite/Credit Research software artifacts must be properly cited (not the same as referenced!) to give credit to authors (evaluation!) Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 19 / 32 Archive, Reference, Describe, Cite and Credit Archive Research software artifacts must be properly archived make sure we can retrieve them (reproducibility) Reference Research software artifacts must be properly referenced make sure we can identify them (reproducibility) Describe Research software artifacts must be properly described make it easy to discover and reuse them (visibility) Cite/Credit Research software artifacts must be properly cited (not the same as referenced!) to give credit to authors (evaluation!) Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 19 / 32 Archive, Reference, Describe, Cite and Credit Archive Research software artifacts must be properly archived make sure we can retrieve them (reproducibility) Reference Research software artifacts must be properly referenced make sure we can identify them (reproducibility) Describe Research software artifacts must be properly described make it easy to discover and reuse them (visibility) Cite/Credit Research software artifacts must be properly cited (not the same as referenced!) to give credit to authors (evaluation!) Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 19 / 32 Archive, Reference, Describe, Cite and Credit Archive Research software artifacts must be properly archived make sure we can retrieve them (reproducibility) Reference Research software artifacts must be properly referenced make sure we can identify them (reproducibility) Describe Research software artifacts must be properly described make it easy to discover and reuse them (visibility) Cite/Credit Research software artifacts must be properly cited (not the same as referenced!) to give credit to authors (evaluation!) Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 19 / 32 Outline 1Introduction 2The software commons: an endangered knowledge 3Avoiding the tragedy of an unattended software commons 4Source code in open science 5Archive and reference 6Describe, cite, credit 7Policy framework 8Call to actions Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 20 / 32 Some old popular approaches A - Since the 1970’s 1990’s .zip or .tar file on: ftp server (e.g. gnu) web page (example) document archive (+ DOI sample) B - Since the 2000’s Rely on software forges institutional/project (e.g. example) free commercial ones: BitBucket, GitHub, GitLab, ... (e.g. parmap) C: a mix of the two Can get no satisfaction... APoor user experience BNo preservation guarantee CCan do so much better Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 20 / 32 Forges are not archives! 2015: the first big bad news Google Code and Gitorious.org shutdown: ~1M endangered repositories broken links in the web of knowledge (my papers too) Big bad news keep coming in summer 2019: BitBucket announces Mercurial VCS sunset july 2020: BitBucket erases 250.000+ repositories (including research software) summer 2022: GitLab.com considers erasing all projects that are inactive for a year In Academia too! 2021: Inria’s old gforge is unplugged... breaks the Opam build chain for OCaml We need a universal archive of software source code: now we have one! Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 21 / 32 Forges are not archives! 2015: the first big bad news Google Code and Gitorious.org shutdown: ~1M endangered repositories broken links in the web of knowledge (my papers too) Big bad news keep coming in summer 2019: BitBucket announces Mercurial VCS sunset july 2020: BitBucket erases 250.000+ repositories (including research software) summer 2022: GitLab.com considers erasing all projects that are inactive for a year In Academia too! 2021: Inria’s old gforge is unplugged... breaks the Opam build chain for OCaml We need a universal archive of software source code: now we have one! Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 21 / 32 Software Heritage is radically different A quick tour as a user designed for source code: Browse (e.g. Apollo 11 excerpt, see also Apollo 11 blog post) like on a developer platform, not a document archive! reference source code: all granularities, using SWHIDs (full specification available online) compare Fig. 1 and conclusions in the 2012 version and the updated version SWHID in a replication experiment guidelines and a full article SWHIDs guarantee integrity like in blockchains demo if time left: 1download a version of a project for a given SWHID 2compute locally the SWHID with swh-identify 3check that the computed id match the given one Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 22 / 32 Software Heritage is radically different A quick tour as a user designed for source code: Browse (e.g. Apollo 11 excerpt, see also Apollo 11 blog post) like on a developer platform, not a document archive! reference source code: all granularities, using SWHIDs (full specification available online) compare Fig. 1 and conclusions in the 2012 version and the updated version SWHID in a replication experiment guidelines and a full article SWHIDs guarantee integrity like in blockchains demo if time left: 1download a version of a project for a given SWHID 2compute locally the SWHID with swh-identify 3check that the computed id match the given one Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 22 / 32 Software Heritage is radically different (cont.) Getting software archived automated harvesting:over 200 million software origins, your researchers’ work may already be there (actually, here)! universal archive: all source code from all platforms (BitBucket, GitHub, GitLab, your own forge, etc.) trigger archival of any code in one click with the updateswh browser extension use webhooks to automatically archive your code (a GitHub action is available too) journals, libraries, open access portals may deposit sourcecode and metadata Example article from IPOL Example article from eLife Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 23 / 32 Software Heritage is radically different (cont.) Getting software archived automated harvesting:over 200 million software origins, your researchers’ work may already be there (actually, here)! universal archive: all source code from all platforms (BitBucket, GitHub, GitLab, your own forge, etc.) trigger archival of any code in one click with the updateswh browser extension use webhooks to automatically archive your code (a GitHub action is available too) journals, libraries, open access portals may deposit sourcecode and metadata Example article from IPOL Example article from eLife Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 23 / 32 A look at some adoption indicators From Melissa Harrison’s OSEC 2022 talk Upcoming on replicabilitystamp.org HAL+SWH in the Open Science software booklet Funding agencies recommendations ANR 2023 guidelines (p. 17) Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 24 / 32 A look at some adoption indicators From Melissa Harrison’s OSEC 2022 talk Upcoming on replicabilitystamp.org HAL+SWH in the Open Science software booklet Funding agencies recommendations ANR 2023 guidelines (p. 17) Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 24 / 32 A look at some adoption indicators From Melissa Harrison’s OSEC 2022 talk Upcoming on replicabilitystamp.org HAL+SWH in the Open Science software booklet Funding agencies recommendations ANR 2023 guidelines (p. 17) Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 24 / 32 Outline 1Introduction 2The software commons: an endangered knowledge 3Avoiding the tragedy of an unattended software commons 4Source code in open science 5Archive and reference 6Describe, cite, credit 7Policy framework 8Call to actions Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 25 / 32 French National plan for Open Science, 2021-2024 SECOND FRENCH PLAN FOR OPEN SCIENCE Generalising open science in France 2021-2024 1 Launch on 6 July 2021 by Frédérique Vidal, Minister for Higher Education, Research and Innovation •Multiplying the levers for change in order to generalise open science practices •Structuring the policy for opening up or sharing research data •New commitments to the opening of source code produced by research •European and international inclusion in the context of the French Presidency of the European Union •Disciplinary and thematic variations: open science policies must be adapted to disciplinary specificities 2 Path Three : Opening up and promoting source code produced by research 3 Recognize and support the dissemination under an open source license of software produced by publicly funded research programmes Highlight the production of source code from higher education, research and innovation Define and promote an open source software policy 789 « The opening of software source code is a major challenge for the reproductibility of scientific results. » « Distribution of software products under open source licence will be preferred. » 4 •Produce a National Charter for Open Source Software coming from higher education, research and innovation •Develop the link between data and software through a network of Chief Data Officers in the various universities and research performing organisations. •Develop the economic models of open source software and make them known within commercialization services •Support Software Heritage and recommend it for the archiving and referencing of source code Define and promote an open source software policy •Create an open source research software prize •Provide greater recognition for software production in the career of researchers, research support staff Recognise source code as a contribution to research •Develop proper coordination between software forges, open publication archives, data repositories and the scientific publishing sector. Build an ecosystem that connects code, data and publications Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 27 / 32 The EOSC SIRS report: Software Source Code and Open Science, 2020 Important policy tool in Open Science (Dec 2020) 9 infrastructures 3 archives 3 open access publishers 3 aggregators recommendations archive in Software Heritage, use SWHID open non profit default to open source for research software "all research software should be made available under an Open Source license by default, and all deviations from this default practice should be properly motivated" See https://doi.org/10.2777/28598 Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 28 / 32 The EOSC SIRS report: Software Source Code and Open Science, 2020 Important policy tool in Open Science (Dec 2020) 9 infrastructures 3 archives 3 open access publishers 3 aggregators recommendations archive in Software Heritage, use SWHID open non profit default to open source for research software "all research software should be made available under an Open Source license by default, and all deviations from this default practice should be properly motivated" See https://doi.org/10.2777/28598 Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 28 / 32 Software in the EOSC Ongoing action in the EOSC Task force on infrastructures for quality research software Foster the development and deployment of tools and services that allow researchers to properly archive, reference, describe with proper metadata, share and reuse research software. Improve the quality of research software, both from the technical and organizational point of view ... Increase recognition to software developers and maintainers of research software ... See the charter of the task force. Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 29 / 32 Software in the EOSC Ongoing action in the EOSC Task force on infrastructures for quality research software Foster the development and deployment of tools and services that allow researchers to properly archive, reference, describe with proper metadata, share and reuse research software. Improve the quality of research software, both from the technical and organizational point of view ... Increase recognition to software developers and maintainers of research software ... See the charter of the task force. Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 29 / 32 The UNESCO recommendations for Open Science (2021) Selection from the recommendations Open Source for Open Science "The source code must be included in the software release and made available on openly accessible repositories and the chosen license must allow modifications, derivative works and sharing under equal or compatible open terms and conditions" Infrastructures "Open science infrastructures should be organized and financed upon an essentially notfor-profit and long-term vision, which enhance open science practices and guarantee permanent and unrestricted access to all, to the largest extent possible." Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 30 / 32 The UNESCO recommendations for Open Science (2021) Selection from the recommendations Open Source for Open Science "The source code must be included in the software release and made available on openly accessible repositories and the chosen license must allow modifications, derivative works and sharing under equal or compatible open terms and conditions" Infrastructures "Open science infrastructures should be organized and financed upon an essentially notfor-profit and long-term vision, which enhance open science practices and guarantee permanent and unrestricted access to all, to the largest extent possible." Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 30 / 32 Outline 1Introduction 2The software commons: an endangered knowledge 3Avoiding the tragedy of an unattended software commons 4Source code in open science 5Archive and reference 6Describe, cite, credit 7Policy framework 8Call to actions Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 31 / 32 Call to action: best practices for ARDC are available... today! Archiving and referencing For all source code used in research (yes, even small scripts!) ensure it is archived in Software Heritage (see save code now) get the proper SWHID for your software (see detailed HOWTO) add it to research articles for reproducibility (see detailed HOWTO) Describing and Citing/Crediting For software you want to put forward (mention in your CV, reports, etc., get citations and credit for it), do the following extra steps: add codemeta.json with description (see the codemeta generator) reference in the HAL portal (french partners, see online HAL documentation) cite software using the biblatex-software package (in CTAN and TeXLive) Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 31 / 32 Call to action: best practices for ARDC are available... today! Archiving and referencing For all source code used in research (yes, even small scripts!) ensure it is archived in Software Heritage (see save code now) get the proper SWHID for your software (see detailed HOWTO) add it to research articles for reproducibility (see detailed HOWTO) Describing and Citing/Crediting For software you want to put forward (mention in your CV, reports, etc., get citations and credit for it), do the following extra steps: add codemeta.json with description (see the codemeta generator) reference in the HAL portal (french partners, see online HAL documentation) cite software using the biblatex-software package (in CTAN and TeXLive) Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 31 / 32 Under the hood Deposit HAL, eLife, IPOL, ... SWORD deposit of software artifacts (2017-...) and metadata (2021) dashboards (global, per user) Save code now ~190.000 requests as of December 2022! from slow (2018) to fast version (2021) updateswh browser extension (August 2022) webhooks for popular forges (end of 2022, to announce) User authentication and authorization keycloak (connection with EOSC AAI will be important soon) Metadata access and software citation will be next GraphQL API (thanks Jayesh), and citation generation widget Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 2 / 12 Outline 9Under the hood 10 HAL and Software Heritage 11 Software Heritage vs. other approaches 12 FAIR software 13 All the source code 14 More on the data model Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 3 / 12 Overview of the Software Heritage / HAL synergy Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 3 / 12 Outline 9Under the hood 10 HAL and Software Heritage 11 Software Heritage vs. other approaches 12 FAIR software 13 All the source code 14 More on the data model Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 4 / 12 Zenodo compare the Parmap library on Zenodo and on Software Heritage connector with InvenioRDM under development as part of FAIRCORE4EOSC Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 4 / 12 Outline 9Under the hood 10 HAL and Software Heritage 11 Software Heritage vs. other approaches 12 FAIR software 13 All the source code 14 More on the data model Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 5 / 12 What about FAIR? (Findable, Accessible, Interoperable, Reusable) FAIR data principles for data in a nutshell: metadata, metadata, metadata all over the place (makes sense for data) But software is not data ... any decent source code comes with reasonable metadata already: good principles have been there for decades the terms interoperability and reusability have precise technical meaning for software, and differ significantly from what is intended by the I and R of FAIR; see the entries for software interoperability and software reusability it is very difficult to achieve these properties even for commercial software developed by multi billion dollars corporations let’s focus on more actionable issues: ARDC a good starting point Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 5 / 12 What about FAIR? (Findable, Accessible, Interoperable, Reusable) FAIR data principles for data in a nutshell: metadata, metadata, metadata all over the place (makes sense for data) But software is not data ... any decent source code comes with reasonable metadata already: good principles have been there for decades the terms interoperability and reusability have precise technical meaning for software, and differ significantly from what is intended by the I and R of FAIR; see the entries for software interoperability and software reusability it is very difficult to achieve these properties even for commercial software developed by multi billion dollars corporations let’s focus on more actionable issues: ARDC a good starting point Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 5 / 12 What about FAIR? (Findable, Accessible, Interoperable, Reusable) FAIR data principles for data in a nutshell: metadata, metadata, metadata all over the place (makes sense for data) But software is not data ... any decent source code comes with reasonable metadata already: good principles have been there for decades the terms interoperability and reusability have precise technical meaning for software, and differ significantly from what is intended by the I and R of FAIR; see the entries for software interoperability and software reusability it is very difficult to achieve these properties even for commercial software developed by multi billion dollars corporations let’s focus on more actionable issues: ARDC a good starting point Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 5 / 12 Outline 9Under the hood 10 HAL and Software Heritage 11 Software Heritage vs. other approaches 12 FAIR software 13 All the source code 14 More on the data model Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 6 / 12 Merkle trees Merkle tree (R. C. Merkle, CRYPTO 1987) Combination of tree hash function Classical cryptographic construction fast, parallel signature of large data structures widely used (e.g., Git, blockchains, IPFS, ...) built-in deduplication Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 10 / 12 Data model Origin + url: str branches Snapshot * * Release directory Revision * ** * Merkle DAG target 1 entries parents snapshots + id: sha1 Directory entries + id: sha1 + id: sha1 + author: str + name: str + message: str + timestamp: datetime + id: sha1 + author: str + message: str + timestamp: datetime Content + id:sha1 Aglobal graph linking together fully deduplicated source code artifact (files, commits, directories, releases, etc.) to the places that distribute them (e.g., Git repositories), providing a unified view on the entire Software Commons. Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 11 / 12 The archive: a (giant) Merkle DAG origin visit https://forge.softwareheritage.org/source/helloworld.git snapshot 1 timestamp Fri Feb 9 12:38:45 2018 +0100 0861db5e… <<Revision>> a3ee21ad… +author = "Stefano Zacchiroli <zack@…>" +message = "add build toolchain …" +timestamp = Thu Feb 8 10:49:29 2018 +0100 +directory: Directory +parents: Revision list <<Directory>> b94a90cd… +entries ".gitignore" "Makefile" "hello.c" <<Content>> 225ae01b… +data = "all: hello\n\n…" <<Revision>> 43ef7dcd… +author = "Stefano Zacchiroli <zack@…>" +message = "add licensing information and README" +timestamp = Thu Feb 8 10:54:09 2018 +0100 +directory: Directory +parents: Revision list <<Directory>> fa8c0908… +entries ".gitignore" "COPYING" "Makefile" "README.md" "hello.c" <<Content>> a1afd006… +data = "…Yet another…" <<Release>> edf82f21… +author = "Stefano Zacchiroli <zack@…>" +name = "1.0" +message = "1.0 release" +timestamp = Thu Feb 8 15:51:00 2018 +0100 +target <<Snapshot>> 0861db5e… +branches HEAD refs/heads/master refs/tags/1.0 Archive content after visit 1 <<Content>> c839dea9… +data = "#include …" <<Directory>> 6ca2e444… +entries "hello.c" <<Revision>> 1886826f… +author = "Stefano Zacchiroli <zack@…>" +message = "implement a trivial …" +timestamp = Thu Feb 8 10:44:35 2018 +0100 +directory: Directory +parents: Revision list = None Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 12 / 12 The archive: a (giant) Merkle DAG origin visit https://forge.softwareheritage.org/source/helloworld.git snapshot 1 https://forge.softwareheritage.org/source/helloworld.git 2 timestamp Fri Feb 9 12:38:45 2018 +0100 Fri Feb 9 13:29:00 2018 +0100 0861db5e… 0861db5e… <<Revision>> a3ee21ad… +author = "Stefano Zacchiroli <zack@…>" +message = "add build toolchain …" +timestamp = Thu Feb 8 10:49:29 2018 +0100 +directory: Directory +parents: Revision list <<Directory>> b94a90cd… +entries ".gitignore" "Makefile" "hello.c" <<Content>> 225ae01b… +data = "all: hello\n\n…" <<Revision>> 43ef7dcd… +author = "Stefano Zacchiroli <zack@…>" +message = "add licensing information and README" +timestamp = Thu Feb 8 10:54:09 2018 +0100 +directory: Directory +parents: Revision list <<Directory>> fa8c0908… +entries ".gitignore" "COPYING" "Makefile" "README.md" "hello.c" <<Content>> a1afd006… +data = "…Yet another…" <<Release>> edf82f21… +author = "Stefano Zacchiroli <zack@…>" +name = "1.0" +message = "1.0 release" +timestamp = Thu Feb 8 15:51:00 2018 +0100 +target <<Snapshot>> 0861db5e… +branches HEAD refs/heads/master refs/tags/1.0 Archive content after visits 1 and 2 <<Content>> c839dea9… +data = "#include …" <<Directory>> 6ca2e444… +entries "hello.c" <<Revision>> 1886826f… +author = "Stefano Zacchiroli <zack@…>" +message = "implement a trivial …" +timestamp = Thu Feb 8 10:44:35 2018 +0100 +directory: Directory +parents: Revision list = None Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 12 / 12 The archive: a (giant) Merkle DAG origin visit https://forge.softwareheritage.org/source/helloworld.git snapshot 1 https://forge.softwareheritage.org/source/helloworld.git 2 timestamp Fri Feb 9 12:38:45 2018 +0100 Fri Feb 9 13:29:00 2018 +0100 https://forge.softwareheritage.org/source/helloworld.git 3Fri Feb 9 15:52:50 2018 +0100 510aa88b… 0861db5e… 0861db5e… <<Revision>> a3ee21ad… +author = "Stefano Zacchiroli <zack@…>" +message = "add build toolchain …" +timestamp = Thu Feb 8 10:49:29 2018 +0100 +directory: Directory +parents: Revision list <<Directory>> b94a90cd… +entries ".gitignore" "Makefile" "hello.c" <<Content>> 225ae01b… +data = "all: hello\n\n…" <<Revision>> 43ef7dcd… +author = "Stefano Zacchiroli <zack@…>" +message = "add licensing information and README" +timestamp = Thu Feb 8 10:54:09 2018 +0100 +directory: Directory +parents: Revision list <<Revision>> c7640e8d… +author = "Stefano Zacchiroli <zack@…>" +message = "move source code to src/\n…" +timestamp = Thu Feb 8 15:26:08 2018 +0100 +directory: Directory +parents: Revision list <<Directory>> fa8c0908… +entries ".gitignore" "COPYING" "Makefile" "README.md" "hello.c" <<Content>> a1afd006… +data = "…Yet another…" <<Directory>> 45f0c078… +entries "COPYING" "Makefile" "README.md" "src" <<Release>> edf82f21… +author = "Stefano Zacchiroli <zack@…>" +name = "1.0" +message = "1.0 release" +timestamp = Thu Feb 8 15:51:00 2018 +0100 +target <<Snapshot>> 0861db5e… +branches HEAD refs/heads/master refs/tags/1.0 <<Snapshot>> 510aa88b… +branches HEAD refs/heads/master refs/heads/doc refs/tags/1.0 Archive content after visits 1 and 2 <<Content>> c839dea9… +data = "#include …" <<Directory>> 6ca2e444… +entries "hello.c" <<Revision>> 1886826f… +author = "Stefano Zacchiroli <zack@…>" +message = "implement a trivial …" +timestamp = Thu Feb 8 10:44:35 2018 +0100 +directory: Directory +parents: Revision list = None ... Archive content after visits 1, 2 and 3 Stefano Zacchiroli [email protected] (CC-BY-SA 4.0) Open Science Best Practices for Source Code Preservation 30 March 2023 12 / 12