Preservare il Software Bene Comune per il Futuro dell'Umanità

Preservare il Software Bene Comune per il Futuro dell’Umanità Stefano Zacchiroli Télécom Paris, Institut Polytechnique de Paris [email protected] 13 Novembre 2022 Biennale Tecnologia Torino, Italy THE GREAT LIBRARY OF SOURC E CODE Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 1 / 26 Outline 1Preface 2The Software Commons 3An Endangered Knowledge 4Avoiding the Tragedy of an Unattended Software Commons 5Researching the Software Commons (highligts) 6Preserving the Software Commons — Present and Future Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 2 / 26 About the speaker Professor of Computer Science, Télécom Paris, Institut Polytechnique de Paris Free/Open Source Software activist (20+ years) Debian Developer & Former 3x Debian Project Leader Former Open Source Initiative (OSI) director Software Heritage co-founder & CTO Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 2 / 26 Outline 1Preface 2The Software Commons 3An Endangered Knowledge 4Avoiding the Tragedy of an Unattended Software Commons 5Researching the Software Commons (highligts) 6Preserving the Software Commons — Present and Future Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 3 / 26 Software is everywhere (and a key mediator) in society Software Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 3 / 26 Free/Open Source Software (FOSS) is everywhere as well Definition (Free Software (1986)) A program is free software if the program’s users have the four essential freedoms: Freedom #0, to run the program, for any purpose Freedom #1, to study how the program works, and change it Freedom #2, to redistribute copies Freedom #3, to improve the program, and release improvements Current estimates: 99% of software products on the market contains at least some free software parts (Synopsis 2020). Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 4 / 26 The Commons and FOSS Definition (Commons) The commons is the cultural and natural resources accessible to all members of a society, including natural materials such as air, water, and a habitable earth. These resources are held in common, not owned privately. Definition (Software Commons) The software commons consists of all computer software which is available at little or no cost and which can be altered and reused with few restrictions. Thus all open source software and all free software are part of the [software] commons. [...] Kranich and Schement (2008); Schweik and English (2012). Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 5 / 26 Software is dual-form knowledge “The source code for a work means the preferred form of the work for making modifications to it." GPL Licence Hello World Program (excerpt of binary) 4004e6: 55 4004e7: 48 89 e5 4004ea: bf 84 05 40 00 4004ef: b8 00 00 00 00 4004f4: e8 c7 fe ff ff 4004f9: 90 4004fa: 5d 4004fb: c3 Program (source code) /* Hello World program */ #include<stdio.h> void main() { printf("Hello World"); } Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 6 / 26 Software is dual-form knowledge “The source code for a work means the preferred form of the work for making modifications to it." GPL Licence Hello World Program (excerpt of binary) 4004e6: 55 4004e7: 48 89 e5 4004ea: bf 84 05 40 00 4004ef: b8 00 00 00 00 4004f4: e8 c7 fe ff ff 4004f9: 90 4004fa: 5d 4004fb: c3 Program (source code) /* Hello World program */ #include<stdio.h> void main() { printf("Hello World"); } Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 6 / 26 Outline 1Preface 2The Software Commons 3An Endangered Knowledge 4Avoiding the Tragedy of an Unattended Software Commons 5Researching the Software Commons (highligts) 6Preserving the Software Commons — Present and Future Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 8 / 26 But where is this commons? many disparate development platforms, with a few dominant players (e.g., GitHub) a myriad places where distribution may happen most of them operated by for-profit companies Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 8 / 26 Software source code is fragile Like all digital information, FOSS is fragile link rot: projects are created, moved around, removed business-driven code loss (e.g., Gitorious, Google Code, Bitbucket) data rot: physical media with legacy software decay If a website disappears you go to the Internet Archive... where do you go if (a repository on) GitHub or GitLab goes away? Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 9 / 26 Software source code is fragile Like all digital information, FOSS is fragile link rot: projects are created, moved around, removed business-driven code loss (e.g., Gitorious, Google Code, Bitbucket) data rot: physical media with legacy software decay If a website disappears you go to the Internet Archive... where do you go if (a repository on) GitHub or GitLab goes away? Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 9 / 26 Calling for preservation: UNESCO UNESCO, Inria, Software Heritage invite 40 international experts meet in Paris ... The call is published on Feb 2019 “[We call to] support efforts to gather and preserve the artifacts and narratives of the history of computing, while the earlier creators are still alive” https://en.unesco.org/foss/paris-call-software-source-code Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 10 / 26 Calling for preservation: UNESCO UNESCO, Inria, Software Heritage invite 40 international experts meet in Paris ... The call is published on Feb 2019 “[We call to] support efforts to gather and preserve the artifacts and narratives of the history of computing, while the earlier creators are still alive” https://en.unesco.org/foss/paris-call-software-source-code Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 10 / 26 Calling for preservation: UNESCO UNESCO, Inria, Software Heritage invite 40 international experts meet in Paris ... The call is published on Feb 2019 “[We call to] support efforts to gather and preserve the artifacts and narratives of the history of computing, while the earlier creators are still alive” https://en.unesco.org/foss/paris-call-software-source-code Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 10 / 26 Calling for preservation: Donald Knuth and Len Shustek Communications of the ACM, February 2021 "Telling historical stories is the best way to teach. It’s much easier to understand something if you know the threads it is connected to." Let’s Not Dumb Down the History of Computer Science Donald E. Knuth, Len Shustek https://doi.org/10.1145/3442377 A unique opportunity most of the creators are still here: we can talk to them! but the clock is ticking... Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 11 / 26 Calling for preservation: Donald Knuth and Len Shustek Communications of the ACM, February 2021 "Telling historical stories is the best way to teach. It’s much easier to understand something if you know the threads it is connected to." Let’s Not Dumb Down the History of Computer Science Donald E. Knuth, Len Shustek https://doi.org/10.1145/3442377 A unique opportunity most of the creators are still here: we can talk to them! but the clock is ticking... Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 11 / 26 Source code history for Open Science Software powers modern research [...] software [...] essential in their fields. Top 100 papers (Nature, 2014) Sometimes, if you dont have the software, you dont have the data Christine Borgman, Paris, 2018 A key pillar: software (source code) The links in the picture are important Nota Bene software may be a tool, a research outcome and a research object access to the source code is essential! Preserving the history of source code is important for reproducibility Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 12 / 26 Outline 1Preface 2The Software Commons 3An Endangered Knowledge 4Avoiding the Tragedy of an Unattended Software Commons 5Researching the Software Commons (highligts) 6Preserving the Software Commons — Present and Future Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 14 / 26 Software Heritage in a nutshell www.softwareheritage.org THE GREAT LIBRA RY OF SO URCE CO D E Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve and share all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 14 / 26 Software Heritage in a nutshell www.softwareheritage.org THE GREAT LIBRA RY OF SO URCE CO D E Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve and share all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 14 / 26 Software Heritage in a nutshell www.softwareheritage.org THE GREAT LIBRA RY OF SO URCE CO D E Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve and share all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 14 / 26 Software Heritage in a nutshell www.softwareheritage.org THE GREAT LIBRA RY OF SO URCE CO D E Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve and share all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 14 / 26 The largest public source code archive, principled bit.ly/swhpaper archive.softwareheritage.org Technology transparency and FOSS replicas all the way down Content (billions!) intrinsic identifiers facts and provenance Organization non-profit multi-stakeholder Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 15 / 26 The largest public source code archive, principled bit.ly/swhpaper archive.softwareheritage.org Technology transparency and FOSS replicas all the way down Content (billions!) intrinsic identifiers facts and provenance Organization non-profit multi-stakeholder Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 15 / 26 A peek under the hood: a global view on the software commons Git loader Mercurial loader Debian source package loader pypi source package loader . . . Software Heritage Archive Merkle DAG + blob storage Loading & deduplication dsc dsc hg hg hg git git git git svn svn svn tar zip software origins Package repos Forges GitHub lister GitLab lister Debian lister PyPi lister . . . Distros ... Scheduling Listing (full/incremental) tar origins snapshots releases revisionsrevisions directoriesdirectories contents Aglobal graph linking together fully deduplicated source code artifact (files, commits, directories, releases, etc.) to the places that distribute them (e.g., Git repositories), providing a unified view on the entire Software Commons. (Size: ~30 B nodes, ~300 B edges, ~1 PiB blobs) Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 16 / 26 A peek under the hood: a global view on the software commons Git loader Mercurial loader Debian source package loader pypi source package loader . . . Software Heritage Archive Merkle DAG + blob storage Loading & deduplication dsc dsc hg hg hg git git git git svn svn svn tar zip software origins Package repos Forges GitHub lister GitLab lister Debian lister PyPi lister . . . Distros ... Scheduling Listing (full/incremental) tar origins snapshots releases revisionsrevisions directoriesdirectories contents Aglobal graph linking together fully deduplicated source code artifact (files, commits, directories, releases, etc.) to the places that distribute them (e.g., Git repositories), providing a unified view on the entire Software Commons. (Size: ~30 B nodes, ~300 B edges, ~1 PiB blobs) Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 16 / 26 Software Heritage intrinsic Identifiers (SWHID) (full spec) An emerging standard in Linux Foundation’s SPDX 2.2 IANA registered, WikiData property P6138 Examples: Apollo 11 AGC excerpt Quake III rsqrt Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 17 / 26 Software provenance and evolution Key findings The amount of original commits in public code doubles every ~30 months and has been doing so for 20+ years; original source code files double every ~22 months It is possible to trace the provenance of source code artifacts at this scale in a compact relational model via the notion of isochrone graphs. Rousseau, Di Cosmo, Zacchiroli Software Provenance Tracking at the Scale of Public Source Code In Empirical Software Engineering, 2020 Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 19 / 26 Diversity, equity, and inclusion Idea Archived commit metadata contains public information that can be mined to study long-term trends of diversity, equity, and inclusion (DEI) traits of the global population of public code contributors. Key findings on the gender gap Male authors contributed 92% of public code commits up to 2019. The ratio of female authors (and their contributions) has grown stably for 15 years reaching for the first time 10% of yearly contributions in 2019. The COVID-19 pandemic has reversed the trend. Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 20 / 26 Diversity, equity, and inclusion (cont.) Key findings on the geographic gap The early decades of public code were dominated by contributions from North America, followed by a period of alternating dominance between North America and Europe. Since then geographic diversity has increased constantly, with raising importance of contributions from Central and South America. The trend of increased female contributions is almost worlwide, with the notable exception of specific regions of Asia were it is either slower or flat. References Zacchiroli. Gender differences in public code contributions: a 50-year perspective. IEEE Software, 2021 Rossi and Zacchiroli. Worldwide gender differences in public code contributions. ICSE SEIS, 2022 Rossi and Zacchiroli. Geographic diversity in public code contributions. MSR 2022 Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 21 / 26 Software Heritage Graph dataset Use case: large scale analyses of the most comprehensive corpus on the development history of free/open source software. Antoine Pietri, Diomidis Spinellis, Stefano Zacchiroli The Software Heritage Graph Dataset: Public software development under one roof MSR 2019: 16th Intl. Conf. on Mining Software Repositories. IEEE preprint: http://deb.li/swhmsr19 Dataset Relational representation of the full graph as a set of tables Available as open data: https://doi.org/10.5281/zenodo.2583978 Chosen as subject for the MSR 2020 Mining Challenge Formats Local use: PostgreSQL dumps, or Apache Parquet files (~1 TiB each) Live usage: Amazon Athena (SQL-queriable), Azure Data Lake Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 22 / 26 Outline 1Preface 2The Software Commons 3An Endangered Knowledge 4Avoiding the Tragedy of an Unattended Software Commons 5Researching the Software Commons (highligts) 6Preserving the Software Commons — Present and Future Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 23 / 26 Academia & policy: growing adoption (selection) HAL software curated deposit workflow Curated Archiving of Research Software Artifacts International Journal of Digical Curation, 2020 Reference archive for swmath.org See code links, e.g. SemiPar package IPOL (image processing) archive (deposit) reference BibLaTeX eLife (life sciences) archive (save code now) reference JTCAM (mechanics) instructions for authors biblatex-software in journal L A T EX class Policy: France National Plan for Open Science and Research Infrastructures Policy: Europe EOSC SIRS report SWHIDs archive Guidelines 1Prepare your public repository README, AUTHORS & LICENSE files 2Save your code http://save.softwareheritage.org/ 3Reference your work (full repository, specific version or code fragment) summary ICMS 2020 Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 23 / 26 Academia & policy: growing adoption (selection) HAL software curated deposit workflow Curated Archiving of Research Software Artifacts International Journal of Digical Curation, 2020 Reference archive for swmath.org See code links, e.g. SemiPar package IPOL (image processing) archive (deposit) reference BibLaTeX eLife (life sciences) archive (save code now) reference JTCAM (mechanics) instructions for authors biblatex-software in journal L A T EX class Policy: France National Plan for Open Science and Research Infrastructures Policy: Europe EOSC SIRS report SWHIDs archive Guidelines 1Prepare your public repository README, AUTHORS & LICENSE files 2Save your code http://save.softwareheritage.org/ 3Reference your work (full repository, specific version or code fragment) summary ICMS 2020 Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 23 / 26 Academia & policy: growing adoption (selection) HAL software curated deposit workflow Curated Archiving of Research Software Artifacts International Journal of Digical Curation, 2020 Reference archive for swmath.org See code links, e.g. SemiPar package IPOL (image processing) archive (deposit) reference BibLaTeX eLife (life sciences) archive (save code now) reference JTCAM (mechanics) instructions for authors biblatex-software in journal L A T EX class Policy: France National Plan for Open Science and Research Infrastructures Policy: Europe EOSC SIRS report SWHIDs archive Guidelines 1Prepare your public repository README, AUTHORS & LICENSE files 2Save your code http://save.softwareheritage.org/ 3Reference your work (full repository, specific version or code fragment) summary ICMS 2020 Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 23 / 26 An international, non profit initiative built for the long term Sharing the vision www.softwareheritage.org/support/testimonials Donors, members, sponsors Diamond sponsor Bronze sponsors Gold sponsors Silver sponsors Platinum sponsors www.softwareheritage.org/support/sponsors Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 24 / 26 You can help! Foster adoption and best practices archive and reference relevant source code (save code now, and deposit) use Software Heritage in research articles, journals, and books rescue and preserve landmark legacy source code with SWHAP and Software Stories Engage with Software Heritage as an individual join the ambassador program, help raise awareness contribute to technical and scientific development Engage with Software Heritage as an organization become a member/sponsor build a Software Heritage mirror Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 25 / 26 Data model Origin + url: str branches Snapshot * * Release directory Revision * ** * Merkle DAG target 1 entries parents snapshots + id: sha1 Directory entries + id: sha1 + id: sha1 + author: str + name: str + message: str + timestamp: datetime + id: sha1 + author: str + message: str + timestamp: datetime Content + id:sha1 Aglobal graph linking together fully deduplicated source code artifact (files, commits, directories, releases, etc.) to the places that distribute them (e.g., Git repositories), providing a unified view on the entire Software Commons. Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 5 / 6 The archive: a (giant) Merkle DAG origin visit https://forge.softwareheritage.org/source/helloworld.git snapshot 1 timestamp Fri Feb 9 12:38:45 2018 +0100 0861db5e… <<Revision>> a3ee21ad… +author = "Stefano Zacchiroli <zack@…>" +message = "add build toolchain …" +timestamp = Thu Feb 8 10:49:29 2018 +0100 +directory: Directory +parents: Revision list <<Directory>> b94a90cd… +entries ".gitignore" "Makefile" "hello.c" <<Content>> 225ae01b… +data = "all: hello\n\n…" <<Revision>> 43ef7dcd… +author = "Stefano Zacchiroli <zack@…>" +message = "add licensing information and README" +timestamp = Thu Feb 8 10:54:09 2018 +0100 +directory: Directory +parents: Revision list <<Directory>> fa8c0908… +entries ".gitignore" "COPYING" "Makefile" "README.md" "hello.c" <<Content>> a1afd006… +data = "…Yet another…" <<Release>> edf82f21… +author = "Stefano Zacchiroli <zack@…>" +name = "1.0" +message = "1.0 release" +timestamp = Thu Feb 8 15:51:00 2018 +0100 +target <<Snapshot>> 0861db5e… +branches HEAD refs/heads/master refs/tags/1.0 Archive content after visit 1 <<Content>> c839dea9… +data = "#include …" <<Directory>> 6ca2e444… +entries "hello.c" <<Revision>> 1886826f… +author = "Stefano Zacchiroli <zack@…>" +message = "implement a trivial …" +timestamp = Thu Feb 8 10:44:35 2018 +0100 +directory: Directory +parents: Revision list = None Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 6 / 6 The archive: a (giant) Merkle DAG origin visit https://forge.softwareheritage.org/source/helloworld.git snapshot 1 https://forge.softwareheritage.org/source/helloworld.git 2 timestamp Fri Feb 9 12:38:45 2018 +0100 Fri Feb 9 13:29:00 2018 +0100 0861db5e… 0861db5e… <<Revision>> a3ee21ad… +author = "Stefano Zacchiroli <zack@…>" +message = "add build toolchain …" +timestamp = Thu Feb 8 10:49:29 2018 +0100 +directory: Directory +parents: Revision list <<Directory>> b94a90cd… +entries ".gitignore" "Makefile" "hello.c" <<Content>> 225ae01b… +data = "all: hello\n\n…" <<Revision>> 43ef7dcd… +author = "Stefano Zacchiroli <zack@…>" +message = "add licensing information and README" +timestamp = Thu Feb 8 10:54:09 2018 +0100 +directory: Directory +parents: Revision list <<Directory>> fa8c0908… +entries ".gitignore" "COPYING" "Makefile" "README.md" "hello.c" <<Content>> a1afd006… +data = "…Yet another…" <<Release>> edf82f21… +author = "Stefano Zacchiroli <zack@…>" +name = "1.0" +message = "1.0 release" +timestamp = Thu Feb 8 15:51:00 2018 +0100 +target <<Snapshot>> 0861db5e… +branches HEAD refs/heads/master refs/tags/1.0 Archive content after visits 1 and 2 <<Content>> c839dea9… +data = "#include …" <<Directory>> 6ca2e444… +entries "hello.c" <<Revision>> 1886826f… +author = "Stefano Zacchiroli <zack@…>" +message = "implement a trivial …" +timestamp = Thu Feb 8 10:44:35 2018 +0100 +directory: Directory +parents: Revision list = None Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 6 / 6 The archive: a (giant) Merkle DAG origin visit https://forge.softwareheritage.org/source/helloworld.git snapshot 1 https://forge.softwareheritage.org/source/helloworld.git 2 timestamp Fri Feb 9 12:38:45 2018 +0100 Fri Feb 9 13:29:00 2018 +0100 https://forge.softwareheritage.org/source/helloworld.git 3Fri Feb 9 15:52:50 2018 +0100 510aa88b… 0861db5e… 0861db5e… <<Revision>> a3ee21ad… +author = "Stefano Zacchiroli <zack@…>" +message = "add build toolchain …" +timestamp = Thu Feb 8 10:49:29 2018 +0100 +directory: Directory +parents: Revision list <<Directory>> b94a90cd… +entries ".gitignore" "Makefile" "hello.c" <<Content>> 225ae01b… +data = "all: hello\n\n…" <<Revision>> 43ef7dcd… +author = "Stefano Zacchiroli <zack@…>" +message = "add licensing information and README" +timestamp = Thu Feb 8 10:54:09 2018 +0100 +directory: Directory +parents: Revision list <<Revision>> c7640e8d… +author = "Stefano Zacchiroli <zack@…>" +message = "move source code to src/\n…" +timestamp = Thu Feb 8 15:26:08 2018 +0100 +directory: Directory +parents: Revision list <<Directory>> fa8c0908… +entries ".gitignore" "COPYING" "Makefile" "README.md" "hello.c" <<Content>> a1afd006… +data = "…Yet another…" <<Directory>> 45f0c078… +entries "COPYING" "Makefile" "README.md" "src" <<Release>> edf82f21… +author = "Stefano Zacchiroli <zack@…>" +name = "1.0" +message = "1.0 release" +timestamp = Thu Feb 8 15:51:00 2018 +0100 +target <<Snapshot>> 0861db5e… +branches HEAD refs/heads/master refs/tags/1.0 <<Snapshot>> 510aa88b… +branches HEAD refs/heads/master refs/heads/doc refs/tags/1.0 Archive content after visits 1 and 2 <<Content>> c839dea9… +data = "#include …" <<Directory>> 6ca2e444… +entries "hello.c" <<Revision>> 1886826f… +author = "Stefano Zacchiroli <zack@…>" +message = "implement a trivial …" +timestamp = Thu Feb 8 10:44:35 2018 +0100 +directory: Directory +parents: Revision list = None ... Archive content after visits 1, 2 and 3 Stefano Zacchiroli Preservare il Software Bene Comune 2022-11-13, Torino 6 / 6