Global Software Health: An Unified View of how our Software Commons is Doing

Global Software Health an Unified View of how our Software Commons is Doing Stefano Zacchiroli Université de Paris & Inria — [email protected], @zacchiro 3 July 2020 SoHeal 2020 (via conf call) THE GREAT LIBRARY OF SOURC E CODE Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 1 / 34 Outline 1Software Health 2Software Commons 3Software Heritage 4Exploring the Software Commons 5Conclusion Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 2 / 34 Software Health Definition (Software Health) One of the hardest research fields to search the Web for. Proof (empirical, trivial). Exhibit: https://www.google.com/search?q=software+health More seriously... The SoHeal community has pioneered the exploration of the notion of Software Health. By now we have evidence of interest in several dimensions of the notion, we have tools & techniques that are routinely used to explore them, and we have been doing that at various scopes. Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 2 / 34 Software Health Definition (Software Health) One of the hardest research fields to search the Web for. Proof (empirical, trivial). Exhibit: https://www.google.com/search?q=software+health More seriously... The SoHeal community has pioneered the exploration of the notion of Software Health. By now we have evidence of interest in several dimensions of the notion, we have tools & techniques that are routinely used to explore them, and we have been doing that at various scopes. Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 2 / 34 Software Health Definition (Software Health) One of the hardest research fields to search the Web for. Proof (empirical, trivial). Exhibit: https://www.google.com/search?q=software+health More seriously... The SoHeal community has pioneered the exploration of the notion of Software Health. By now we have evidence of interest in several dimensions of the notion, we have tools & techniques that are routinely used to explore them, and we have been doing that at various scopes. Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 2 / 34 Software Health — dimensions What are we looking at Several dimensions have been explored thus far, e.g.: software evolution and "liveliness" quality (cf. SoHeal 2019 keynote by Jesus M. Gonzalez-Barahona) community both static structure and dynamics over time (non-exhaustive list) Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 3 / 34 Software Health — tools & techniques How we are exploring the topic classic software evolution & MSR techniques quantitative analysis (stats !) qualitative analysis e.g., interviews, ethnography, Delphi method community metrics & their standardization (cf. CHAOSS) raising awareness in relevant communities: FOSS + scholars the SoHeal workshop series! Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 4 / 34 Software Health — scope How far are we looking 1a single project 2a set of inter-dependent projects e.g., a specific framework with plugins, a software stack, etc. also a community of contributors working on said projects 3an ecosystem e.g., Debian, PyPI, NPM, etc. Going further can we go further in terms of software health scope? how far? is there a meaningful notion of "global software health"? if there is, which the tools can we use to explore global software health? if they exist and are practical, what is the current status of global software health? Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 5 / 34 Software Health — scope How far are we looking 1a single project 2a set of inter-dependent projects e.g., a specific framework with plugins, a software stack, etc. also a community of contributors working on said projects 3an ecosystem e.g., Debian, PyPI, NPM, etc. Going further can we go further in terms of software health scope? how far? is there a meaningful notion of "global software health"? if there is, which the tools can we use to explore global software health? if they exist and are practical, what is the current status of global software health? Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 5 / 34 Outline 1Software Health 2Software Commons 3Software Heritage 4Exploring the Software Commons 5Conclusion Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 9 / 34 Software Heritage in a nutshell softwareheritage.org THE GREAT LIBRA RY OF SOURCE CO DE Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 9 / 34 Software Heritage in a nutshell softwareheritage.org THE GREAT LIBRA RY OF SOURCE CO DE Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 9 / 34 Software Heritage in a nutshell softwareheritage.org THE GREAT LIBRA RY OF SOURCE CO DE Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 9 / 34 Software Heritage in a nutshell softwareheritage.org THE GREAT LIBRA RY OF SOURCE CO DE Collect, preserve and share all software source code Preserving our heritage, enabling better software and better science for all Reference catalog find and reference all software source code Universal archive preserve all software source code Research infrastructure enable analysis of all software source code Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 9 / 34 An international, non profit initiative built for the long term Sharing the vision www.softwareheritage.org/support/testimonials Donors, members, sponsors Platinum sponsors Silver sponsors Bronze sponsors Gold sponsor www.softwareheritage.org/support/sponsors Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 10 / 34 Archiving goals Targets: VCS repositories & source code releases (e.g., tarballs) We DO archive file content (= blobs) revisions (= commits), with full metadata releases (= tags), ditto where (origin) & when (visit) we found any of the above ... in a VCS-/archive-agnostic canonical data model We DON’T archive homepages, wikis BTS/issues/code reviews/etc. mailing lists Long term vision: play our part in a "semantic wikipedia of software" Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 11 / 34 Data flow deb deb hg hg hg git git git git svn svn svn pypi pypi software origins Package repos Software Heritage Archive Forges GitHub lister GitLab lister Debian lister Git loader Mercurial loader Debian source package loader PyPI lister PyPI loader Merkle DAG + blob storage . . . . . . Distros ... Scheduling Listing (full/incremental) Loading & deduplication Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 12 / 34 Merkle trees Merkle tree (R. C. Merkle, CRYPTO 1987) Combination of tree hash function Classical cryptographic construction fast, parallel signature of large data structures widely used (e.g., Git, blockchains, IPFS, ...) built-in deduplication Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 13 / 34 Merkle trees Merkle tree (R. C. Merkle, CRYPTO 1987) Combination of tree hash function Classical cryptographic construction fast, parallel signature of large data structures widely used (e.g., Git, blockchains, IPFS, ...) built-in deduplication Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 13 / 34 Outline 1Software Health 2Software Commons 3Software Heritage 4Exploring the Software Commons 5Conclusion Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 17 / 34 Early days We are in the early days of full-scale explorations of the entire software commons, for both software health and other research or practical needs. We are also not yet capable of performing analyses at such scale, due to a lack of resources (including time!) and/or appropriate tools and techniques. In the following I’ll review some related work: a large-scale dataset encompassing a decent chunk of the software commons atechnique to exploit such dataset on a budget a long-term exploration of the growth rate of the software commons Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 17 / 34 Software Heritage Graph dataset Use case: large scale analyses of the most comprehensive corpus on the development history of free/open source software. Antoine Pietri, Diomidis Spinellis, Stefano Zacchiroli The Software Heritage Graph Dataset: Public software development under one roof MSR 2019: 16th Intl. Conf. on Mining Software Repositories. IEEE preprint: http://deb.li/swhmsr19 Dataset Relational representation of the full graph as a set of tables Available as open data: https://doi.org/10.5281/zenodo.2583978 Chosen as subject for the MSR 2020 Mining Challenge Formats Local use: PostgreSQL dumps, or Apache Parquet files (~1 TiB each) Live usage: Amazon Athena (SQL-queriable), Azure Data Lake (soon) Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 18 / 34 Sample query — most frequent first commit words SELECT COUNT(*)AS c,word FROM ( SELECT LOWER(REGEXP_EXTRACT(FROM_UTF8( message), '^\w+')) AS word FROM revision) WHERE word != '' GROUP BY word ORDER BY COUNT(*)DESC LIMIT 5; Count Word 71 338 310 update 64 980 346 merge 56 854 372 add 44 971 954 added 33 222 056 fix Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 19 / 34 Sample query — most frequent first commit words SELECT COUNT(*)AS c,word FROM ( SELECT LOWER(REGEXP_EXTRACT(FROM_UTF8( message), '^\w+')) AS word FROM revision) WHERE word != '' GROUP BY word ORDER BY COUNT(*)DESC LIMIT 5; Count Word 71 338 310 update 64 980 346 merge 56 854 372 add 44 971 954 added 33 222 056 fix Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 19 / 34 Sample query — fork and merge arities Fork arity i.e., how often is a commit based upon? SELECT fork_deg,count(*)FROM ( SELECT id,count(*)AS fork_deg FROM revision_history GROUP BY id)t GROUP BY fork_deg ORDER BY fork_deg; 0.1 1 10 100 1000 10000 100000 1x106 1x107 1x108 1x109 1x1010 0.1 1 10 100 1000 10000 100000 1x106 1x107 Number of nodes Degree Merge arity i.e., how large are merges? SELECT merge_deg,COUNT(*)FROM ( SELECT parent_id,COUNT(*)AS merge_deg FROM revision_history GROUP BY parent_id)t GROUP BY deg ORDER BY deg; 0.1 1 10 100 1000 10000 100000 1x106 1x107 1x108 1x109 1x1010 0.1 1 10 100 1000 10000 100000 1x106 1x107 Number of nodes Degree Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 20 / 34 Sample query — fork and merge arities Fork arity i.e., how often is a commit based upon? SELECT fork_deg,count(*)FROM ( SELECT id,count(*)AS fork_deg FROM revision_history GROUP BY id)t GROUP BY fork_deg ORDER BY fork_deg; 0.1 1 10 100 1000 10000 100000 1x106 1x107 1x108 1x109 1x1010 0.1 1 10 100 1000 10000 100000 1x106 1x107 Number of nodes Degree Merge arity i.e., how large are merges? SELECT merge_deg,COUNT(*)FROM ( SELECT parent_id,COUNT(*)AS merge_deg FROM revision_history GROUP BY parent_id)t GROUP BY deg ORDER BY deg; 0.1 1 10 100 1000 10000 100000 1x106 1x107 1x108 1x109 1x1010 0.1 1 10 100 1000 10000 100000 1x106 1x107 Number of nodes Degree Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 20 / 34 Sample query — ratio of commits performed during weekends WITH revision_date AS (SELECT FROM_UNIXTIME(date /1000000)AS date FROM revision) SELECT yearly_rev.year AS year, CAST(yearly_weekend_rev.number AS DOUBLE) /yearly_rev.number *100.0AS weekend_pc FROM (SELECT YEAR(date)AS year,COUNT(*)AS number FROM revision_date WHERE YEAR(date)BETWEEN 1971 AND 2018 GROUP BY YEAR(date))AS yearly_rev JOIN (SELECT YEAR(date)AS year,COUNT(*)AS number FROM revision_date WHERE DAY_OF_WEEK(date)>= 6 AND YEAR(date)BETWEEN 1971 AND 2018 GROUP BY YEAR(date))AS yearly_weekend_rev ON yearly_rev.year =yearly_weekend_rev.year ORDER BY year DESC; Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 21 / 34 Sample query — ratio of commits performed during weekends (cont.) Year Weekend Total Weekend percentage 2018 15130065 78539158 19.26 2017 33776451 168074276 20.09 2016 43890325 209442130 20.95 2015 35781159 166884920 21.44 2014 24591048 122341275 20.10 2013 17792778 88524430 20.09 2012 12794430 64516008 19.83 2011 9765190 48479321 20.14 2010 7766348 38561515 20.14 2009 6352253 31053219 20.45 2008 4568373 22474882 20.32 2007 3318881 16289632 20.37 2006 2597142 12224905 21.24 2005 2086697 9603804 21.72 2004 1752400 7948104 22.04 2003 1426033 6941593 20.54 2002 1159294 5378538 21.55 2001 849905 4098587 20.73 2000 2091770 4338842 48.21 1999 438540 2026906 21.63 1998 311888 1430567 21.80 1997 263995 1129249 23.37 1996 192543 795827 24.19 1995 176270 670417 26.29 1994 137811 581563 23.69 1993 169767 697343 24.34 1992 74923 422068 17.75 1991 92782 484547 19.14 1990 113201 340489 33.24 1989 31742 182325 17.40 1988 44983 206275 21.80 1987 27892 146157 19.08 1986 54200 237330 22.83 1985 75595 306564 24.65 1984 26391 95506 27.63 1983 89776 370687 24.21 1982 51524 191933 26.84 1981 32995 123618 26.69 1980 31832 133733 23.80 1979 20943 175164 11.95 1978 3773 33677 11.20 1977 4783 19376 24.68 1976 1907 7048 27.05 1975 2089 26579 7.85 1974 2095 14290 14.66 1973 2988 15580 19.17 1972 1755 6552 26.78 1971 1723 6125 28.13 Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 22 / 34 Sample query — average size of the most popular file types SELECT suffix, ROUND(COUNT(*)*100 /1e6)AS Million_files, ROUND(AVG(length)/1024)AS Average_k_length FROM (SELECT length,suffix FROM -- File length in joinable form (SELECT TO_BASE64(sha1_git)AS sha1_git64,length FROM content )AS content_length JOIN -- Sample of files with popular suffixes (SELECT target64,file_suffix_sample.suffix AS suffix FROM -- Popular suffixes (SELECT suffix FROM ( SELECT REGEXP_EXTRACT(FROM_UTF8(name), '\.[^.]+$')AS suffix FROM directory_entry_file)AS file_suffix GROUP BY suffix ORDER BY COUNT(*)DESC LIMIT 20 )AS pop_suffix JOIN -- Sample of files and suffixes (SELECT TO_BASE64(target)AS target64, REGEXP_EXTRACT(FROM_UTF8(name), '\.[^.]+$')AS suffix FROM directory_entry_file TABLESAMPLE BERNOULLI(1)) AS file_suffix_sample ON file_suffix_sample.suffix =pop_suffix.suffix) AS pop_suffix_sample ON pop_suffix_sample.target64 =content_length.sha1_git64) GROUP BY suffix ORDER BY AVG(length)DESC; Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 23 / 34 Compression efficiency (space) Forward graph total size 91 GiB bits per edge 4.91 compression ratio 15.8% Backward graph total size 83 GiB bits per edge 4.49 compression ratio 14.4% Operating cost The structure of a full bidirectional archive graph fits in less than 200 GiB of RAM, for a hardware cost of ~300 USD. Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 30 / 34 Compression efficiency (time) Benchmark — Full BFS visit (single thread) Forward graph wall time 1h48m throughput 1.81 M nodes/s (553 ns/node) Backward graph wall time 3h17m throughput 988 M nodes/s (1.01 µs/node) Benchmark — Edge lookup random sample: 1 B nodes (8.3% of entire graph); then enumeration of all successors Forward graph visited edges 13.6 B throughput 12.0 M edges/s (83 ns/edge) Backward graph visited edges 13.6 B throughput 9.45 M edges/s (106 ns/edge) Note how edge lookup time is close to DRAM random access time (50-60 ns). Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 31 / 34 Discussion Incrementality compression is not incremental, due to the use of contiguous integer ranges but the graph is append-only, so... ...based on expected graph growth rate it should be possible to pre-allocate enough free space in the integer ranges to support amortized incrementality (future work) In-memory v. on-disk the compressed in-memory graph structure has no attributes usual design is to exploit the 0..N-1 integer ranges to memory map node attributes to disk for efficient access works well for queries that does graph traversal first and "join" node attributes last; ping-pong between the two is expensive edge attributes are more problematic Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 32 / 34 Discussion Incrementality compression is not incremental, due to the use of contiguous integer ranges but the graph is append-only, so... ...based on expected graph growth rate it should be possible to pre-allocate enough free space in the integer ranges to support amortized incrementality (future work) In-memory v. on-disk the compressed in-memory graph structure has no attributes usual design is to exploit the 0..N-1 integer ranges to memory map node attributes to disk for efficient access works well for queries that does graph traversal first and "join" node attributes last; ping-pong between the two is expensive edge attributes are more problematic Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 32 / 34 Original content growth 50 years of software commons history. 50 M projects, 4 B blobs, 1 B commits (Software Heritage snapshot, Feb 2018) original artifacts explored over time, after deduplication evidence of exponential growth: original commits doubles every 30 months; blobs every 22 months; original blobs per commit doubles every 7 years Roberto Di Cosmo, Guillaume Rousseau, Stefano Zacchiroli Software Provenance Tracking at the Scale of Public Source Code Empirical Software Engineering, 2020 Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 33 / 34 Outline 1Software Health 2Software Commons 3Software Heritage 4Exploring the Software Commons 5Conclusion Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 34 / 34 Wrapping up the notion of software health is shaping up nicely, with several dimensions to it and more and more established tools and techniques global software health, i.e., the study of software health at the scale of the full software commons is an open challenge that requires exhaustive code libraries, tools, and techniques Software Heritage is one such library, containing a significant span of the software commons; tools and techniques to analyze it are now badly needed meanwhile, the software commons seems to be doing well in terms of growth; let’s dig it further to assess its health! Contacts Stefano Zacchiroli / [email protected] / @zacchiro / @[email protected] Stefano Zacchiroli Global Software Health 3 July 2020, SoHeal 34 / 34