Mitigating Query Rot: Using snapquery for Sustainable SPARQL Query Set Management

Mitigating Query Rot: Using snapquery for Sustainable SPARQL Query Set Management Wolfgang Fahl1,2, Christoph Lange1,3, Tim Holzheim1, and Stefan Decker1,3 1RWTH Aachen University, Computer Science i5, Aachen, Germany 2BITPlan GmbH, Willich, Germany 3Fraunhofer FIT, Sankt Augustin, Germany Abstract. Will the “cats” Wikidata SPARQL query example still work in the future? While link rot has been a known issue, the Query Rot problem has not been investigated much in the past. We introduce an approach for quality assessment and refactoring of query sets and evaluate its implementation for SPARQL. By applying information hiding and dependency inversion principles, we hide the details of the query and introduce the snapquery SPARQL query endpoint middleware to maintain expected query behavior independently of technical details, context, and even, as a future option, the query language. “snapquery cats” is designed to work consistently in the future no matter what changes. This approach enables the swapping of endpoints, conceals the complexity of (federated) queries, and manages SPARQL query sets by making the queries FAIR first-class citizens of the Knowledge Graph infrastructure. The increased abstraction aligns well with state-of-the-art Artificial Intelligence approaches that allow the use of natural language input to generate query sets. This novel, systematic, and semi-automatic approach is generally useful in most knowledge graph management scenarios. It allows for the gathering of query metadata from real-world environments on the fly, supporting the creation of test suites, benchmarks, challenges, dashboards, and other analytical applications for query performance and health monitoring. Our approach contributes to advancing knowledge engineering by bridging gaps between knowledge graphs, software engineering, and large language models. We reproduce use cases of the Scholia, QLever and general Wikidata projects to demonstrate functionality and measure non-functional quality improvements over those projects. We conclude that advancing Knowledge Graph management requires 1. introducing queries as FAIR first-class citizens by developing platformindependent named parameterized queries (thus confirming a 2019 proposal), 2. structured, implementation-independent error messages, and 3. shifting to higher abstraction levels that support human and machine generated general text input. We suggest corresponding improvements to the SPARQL standard. Keywords: SPARQL ·Query Rot ·Knowledge Graph Management ·Quality Assessment ·Refactoring ·snapquery ·Benchmarking ·Sustainability 2 Wolfgang Fahl, Christoph Lange, Tim Holzheim, and Stefan Decker 1 Introduction Will the “cats” Wikidata SPARQL query example still work in the future? Listing 1.1 shows the SPARQL code and Table 1 the first six result rows4. Listing 1.1: SPARQL Query for Cats # "Cats" example SPARQL query # https :// www. wikidata . org # / wiki / Wikidata : SPARQL_query_service / queries / examples # Cats SELECT ? item ? itemLabel WHERE { ?item wdt:P31 wd: Q146 . # Must be a cat SERVICE wikibase : label { bd:serviceParam wikibase:language "[ AUTO_LANGUAGE ], en ". } } Table 1: Cats query result item itemLabel http://www.wikidata.org/entity/Q378619 CC http://www.wikidata.org/entity/Q498787 Muezza http://www.wikidata.org/entity/Q677525 Orangey http://www.wikidata.org/entity/Q851190 Mrs. Chippy http://www.wikidata.org/entity/Q893453 Unsinkable Sam http://www.wikidata.org/entity/Q1050083 Catmando ... ... The Cats query most likely will not run in a few years due to Query Rot. The query might not run on endpoints other than the currently in-use “Blazegraph” SPARQL endpoint provided as the Wikidata Query Service by the Wikimedia Foundation. Semantic web technologies that utilize RDF and SPARQL (see Section 3.2)) are commonly used as Knowledge Graph infrastructure. A prevalent issue analogous to link rot (see Section 3.3) in web content is Query Rot for the necessary data retrieval components. Query Rot refers to the gradual deterioration of query validity over time due to changes in the Query Execution Context (QEC) (see Section 4). The crucial reliability, robustness, and other non functional quality aspects of KG systems are challenged by Query Rot. Query Rot is mitigated by repeatedly refactoring queries – an error-prone, manual and time-consuming task, which can only be done by knowledge engi4The table was generated with the command line snapquery –queryName cats –limit 6 –format latex Mitigating Query Rot with snapquery 3 neering experts. snapquery is a tool and method that facilitates the refactoring of queries using a systematic approach, which can be automated and accessed via (standard compliant) APIs as a middleware. Queries are treated as first-class FAIR [35] citizens having a PID5in the form of a domain/namespace/name combination with a link to the original query source(s). The principle of information hiding is applied via the Dependency Inversion Principle (see Section 3.1) (part of the SOLID approach in software engineering). Queries are divided into ablack-box abstract part that only provides a query signature, consisting of the query PID and its parameters and variables and the white-box part detailing the Query Execution Context. The snapquery separation of concerns targets different audiences - the blackbox part for general end users and system integrators and the white-box part for scholars, developers, people running the infrastructure and others with an interest in all graphic detail. snapquery was introduced at the Wikimedia 2024 Hackathon [6], motivated by needs of the Scholia community arising from the upcoming Wikidata graph split (see Section 2.1). snapquery allows for faster systematic and semi-automatic refactoring of queries by assigning metadata about functional and non-functional aspects to each named query and grouping sets of queries by domain and namespace. The knowledge about whether a query (or set of queries) works at all in a given Query Execution Context 2.2 and, e.g., how fast and reliable it runs is stored as metadata and used as a basis for standard refactoring and user feedback activities. The metadata is gathered proactively and automatically and annotated and analyzed with Large-Language-Model (LLM) support. We aim at changing the mindset of KG project stakeholders working with SPARQL query sets. Our approach has potential value in mitigating software architecture and engineering challenges that plague current knowledge engineering projects. This paper is structured around three primary use cases which are presented in section 2.1: – Wikidata – SPARQL examples, tutorials and usage – Scholia – Scholarly publishing – QLever – SPARQL engine development Section 2 introduces and defines the term Query Rot based on a black-box and white-box definition of Query Execution Context targeted to the audiences mentioned above. Section 3 contains prior work and work we build on. Section 4 presents snapquery as a method and tool for mitigating Query Rot. Section 5 details the implementation and Section 6 the evaluation of snapquery. Section 7 concludes and suggests future work. Further background material and research documentation is available at Category:snapquery [8] in our research wiki. 5FAIR and PIDs are well explained in our CEUR-WS semantification work [10] 4 Wolfgang Fahl, Christoph Lange, Tim Holzheim, and Stefan Decker 2 Query Rot 2.1 Motivating use cases Table 2: Motivating Usecases and Query Sets Usecase Namespace@Domain Name Example # Queries Wikidata examples Cats 302 @wikidata.org short-url Japanese_Libraries_Details 100 @wikidata.org federated-queries Editors of the WOP 2014 11 @bitplan.com Subtotal 413 Scholia named-queries author_list-of-publications 373 @scholia.toolforge.org challenge AllVolumes 28 @ceur-ws.org WikidataThesisToolkit LSE-doctoral-theses 27 @wikidata.org Subtotal 428 QLever issues-wikidata Issue858-query1 181 @qlever.cs.uni-freiburg.de performance-dblp All papers published in SIGIR 6 @qlever.cs.uni-freiburg.de examples PublicationTypes 5 @dblp.org Subtotal 192 Total 1033 Table 2 shows the relevant use cases and query sets that have motivated the snapquery approach and tool and that are presented in this work. Wikidata SPARQL examples and usage The Wikidata SPARQL Service examples wiki page (https://www.wikidata.org/wiki/Wikidata:SPARQL_query_ service/queries/) has accumulated over 300 queries since 2016, illustrating Wikidata’s usage. These queries are expected to work against the Wikidata service endpoint. However, this work reveals that this is not always the case and introduces methods to systematically analyze the reasons. Wikidata’s query service (https://query.wikidata.org/) offers short URLs for entered queries. A random set of 100 such queries has been included for investigation in this study and automatically annotated with an LLM (see Section 6.4). Additionally, a set of manually curated federated queries completes the Wikidata use case query sets, totaling 413 queries. Mitigating Query Rot with snapquery 5 Scholia and Scholarly Publishing Scholia(https://scholia.toolforge. org/) is a project allowing users to search, browse, and analyze scholarly publishing data curated in the Wikidata knowledge graph [24]. It uses named parameterized queries, with Python and JavaScript as programming languages, SPARQL for querying, and Jinja Templates for query parameterization. Scholia faces challenges due to Wikidata’s size limitations. The Blazegraph SPARQL engine backing Wikidata can hold up to 4 Terabytes, which is insufficient for the vast amount of scholarly publishing data available [11]. The Wikimedia Foundation’s decision to split the graph and migrate scholarly data to its own knowledge graph potentially invalidates all 373 current Scholia queries. The query sets of 428 queries for the Scholia use case comes from multiple sources: 373 named queries extracted from Scholia’s GitHub repositorym 28 Semantic Publishing Challenge [18] queries presented in our research Semantic MediaWiki, and WikidataThesisToolkit [36] queries documented on a wiki page. QLever SPARQL Engine Development QLever, developed at the University of Freiburg, is a high-performance SPARQL Engine written in C++ [3]. It is a potential replacement for Blazegraph as Wikidata’s main SPARQL engine. QLever is an open-source project [1] but is not yet feature-complete. We proposed using Scholia queries as a test suite for QLever6. This work extends that idea by demonstrating how to construct a testsuite from SPARQL queries extractable from GitHub issues. A typical issue, such as the issue #896 CONCAT implementation7, follows a standard situation/action/expected result format, which can be used for systematic testing and development. The query set of 192 queries for the QLever use case is derived from queries extracted from QLever’s GitHub issues, performance queries related to DBLP documented on a wiki page in the GitHub repository, and example queries from the QLever-UI portal for the DBLP example dataset. 2.2 Definitions We present two complementary definitions of Query Execution Context (QEC): Definition 2 – a concrete/white-box definition for scholars, developers, people running the infrastructure and others with an interest in all graphic detail. Definition 3 – an abstract/black-box definition for general end users, system integrators and the general public. The white-box definition illustrates the high dimensionality of the solution space and can not be elaborated in all detail here, while the black-box definition describes the simplified problem space that we intend to focus on in the core Query Rot Definition 4. Definition 1 (Named Parameterized Query). A Named Parameterized Query Qis a tuple (N, P, V ), where: 6https://github.com/ad-freiburg/qlever/issues/859 7https://github.com/ad-freiburg/qlever/issues/896 6 Wolfgang Fahl, Christoph Lange, Tim Holzheim, and Stefan Decker –Nis the unique name of the query, structured as name−−namespace@domain –P={(p1:t1),...,(pk:tk)}is the set of typed input parameters –V={(v1:t1),...,(vm:tm)}is the set of typed output variables The unique name convention is inspired by the Java Naming Convention [25]. Definition 2 (Query Execution Context (concrete/white-box)). A Query Execution Context QEC is a function QEC :QS ×G×B×EE ×L→R, where: –QS ={qi|i∈I}is a set of Named Parameterized Queries, where each qi is of type Qas defined in Definition 1, and Iis an index set, –G= (V, E)is the knowledge graph being queried, –B={b1, b2, . . . , bk}represents a set of boundary conditions, –EE ={e1, e2, . . . , em}represents a set of execution environment parameters, –Lrepresents the query language standard and feature set, –R= (S, T, M)is the result, where Sis an output stream, Tis the content type, and Mis metadata about the query execution. Definition 3 (Query Execution Context (abstract/black-box)). A Query Execution Context QEC is a function QEC :QS ×G→R, where: –QS ={q1, q2, ..., qn}is a set of NamedQueries as defined in Definition 1, –Gis the knowledge graph being queried, –R= (S, T, M)is the result, where Sis an output stream, Tis the content type, and Mis metadata about the query execution. The black-box definition reduces the complexity of the problem space to QS ×KG, while the white-box definition exposes the full dimensionality of the solution space as QS ×G×B×E×L. The abstraction enables more effective knowledge engineering by simplifying the user’s conceptual model while allowing for complex optimizations in the implementation and providing standardized APIs independent of the hidden details. An example of a Query Execution Context in practice would consist of QS being the Scholia Queryset “[email protected]” (see Section 2.1) consisting of more than 373 queries in the context of Wikidata as the knowledge graph G. Boundary conditions Binclude legal rules such as data protection and copyright laws, and limits set by organizational rules. The execution environment EE describes the Wikimedia Foundation’s data center running a cluster of Blazegraph instances and the 1 minute time out for the public Wikidata Query Service. Lrepresents the SPARQL 1.1 language with Blazegraph extensions. For the specific query author_events––[email protected], we have: N=author_events––[email protected] P={(author :Q80 )}– Q80 is the Wikidata ID for Tim Berners-Lee V={(Date: xsd:dateTime),(Event: IRI),(EventLabel: xsd:string),(EventUri: IRI),(Roles: xsd:string),(Locations: xsd:string)} An example output row for R= (S, T, M): Mitigating Query Rot with snapquery 7 S: HTML table containing a row: Date: 2009-10-25, Event: http://www.wikidata.org/entity/Q48026503, EventLabel: The 8th International Semantic Web Conference, EventUri: /event/Q48026503, Roles: author, Locations: Washington, D.C. T: HTML M: {execution time: 0.4 seconds, result count: 25} Definition 4 (Query Rot). Query Rot is a phenomenon that occurs when a Query Execution Context QEC, which previously produced satisfactory results for all queries q∈QS, experiences degradation or failure of query executions due to changes in underlying system components, despite no relevant changes to the query set QS or knowledge graph G. Query Rot of a QEC manifests when for some q and some input parameter set of QS –qfails to execute, –qreturns unexpected, or inconsistent results, or –the metadata of an execution qfails to meet specified functional or nonfunctional criteria or boundary conditions. In the context of the white-box definition, Query Rot typically arises from unaccounted changes in: –L: the query language standard or feature set, –EE: the execution environment, –B: the implicit or explicit boundary conditions, or –G: the knowledge graph data or schema. From the black-box perspective, Query Rot is observed as a change in the relationship between inputs (QS and KG) and output R, without apparent changes to these visible components. 3 Background and Related Work 3.1 Software Engineering Principles Parnas introduced Information Hiding as a core principle of software engineering in his paper “On the Criteria to be Used in Decomposing Systems into Modules” [26]. He proposes modularization as a strategy to improve software quality. Parnas argues that instead of the conventional flowchart-based structure, a system decomposition approach should be applied: based on “information hiding”, it groups modules by hidden design decisions rather than processing steps. Martin proposed the Dependency Inversion Principle [22] stating, “Depend upon Abstractions. Do not depend upon concretions”. This principle is a key strategy for achieving systematic information hiding. It advocates for the use of black-box intermediary interfaces to abstract away technical white-box implementation details, thereby reducing coupling between components and enhancing modularity, flexibility, and maintainability of software systems – thus reducing development time in the long run. A comprehensive description of the state-of-the art of Continuous Integration (CI) and Continuous Delivery (CD) is given by Van Merode [23] 8 Wolfgang Fahl, Christoph Lange, Tim Holzheim, and Stefan Decker 3.2 SPARQL SPARQL (SPARQL Protocol and RDF Query Language) is the standard query language for RDF [5] data, recommended by the W3C. Since its introduction in 2008 [28], SPARQL has undergone several revisions, with SPARQL 1.1 being the current version as of 2013 [16]. SPARQL allows for complex queries on linked RDF datasets by mapping variables to solutions represented as multisets of triples. Note that the results might not necessarily be “truly tabular” [13]. SPARQL has been the focus of research for the past decades, prominently featured in conference series such as the International Semantic Web Conference (ISWC), the Extended Semantic Web Conference (ESWC), the World Wide Web Conference (WWW), and the International Conference on Knowledge Engineering and Knowledge Management (EKAW). The traditional research approach applies a “white-box” view where the emphasis is on the technical and theoretical underpinnings and all details are explored such as in ontology design having researchers and developers as a target group. The formal analysis of the semantics of the SPARQL language by Perez et al. [27] is an example. It establishes the algebraic foundation of SPARQL based on graph patterns and RDF triples, contrasting with relational algebra, which operates on tables and lacks native support for optional data and flexible pattern matching. The “white-box” view with end-users and system-integrators as a target group is somewhat underrepresented. Paul Warren et al.’s work [32,31,33,34] is a notable exception. His work is rooted in the semantification of the BT Library, which is related to our scholarly publishing use case (Section 2.1). According to Warren [34], query expert knowledge is rare – 74% of his study participants claimed to have no knowledge at all. In a 2018 study [33], the minority of participants were end-users and large databases with billions of triples were in the majority to be targeted by query sets. Timeouts were named as the main difficulty with SPARQL. Johannes Lorey proposes Latency, Throughput, Execution Time of Joins as metrics for the Quality of Query Execution Contexts [19]. He also worked on discovering query templates from query logs [21] which supports the proposal of introducing named parameterized queries. His PhD thesis “What’s in a Query: Analyzing, Predicting, and Managing Linked Data Access” [20] elaborates on the details. The SPARQL standard does not call for enforcing structured error messages on failure: “The response body of a failed query request is implementation defined. Implementations may use HTTP content negotiation to provide humanreadable or machine-processable (or both) information about the failed query request.” [15] The introduction of parameterized queries to SPARQL was proposed as an addition to the W3C Standard by Vladimir Alexiev [2] in 2019. As of 2024 there are still only implementation specific solutions. Mitigating Query Rot with snapquery 9 3.3 Link Rot Link rot describes cases where hyperlinks are getting invalid over time [29]. This happens when the target resource has been relocated to a new address (“reroute”) or has become permanently unavailable “dead”. A study on “link decay” by Hennessy for the time span 1999–2010 found that the median lifespan of web pages was 9.3 years [17]. Link rot breaks KG functionality and leads to frustrations, particularly in scholarly publishing contexts where citations rely on links [4]. One cause of Link rot is an orphaned responsibility for maintaining a link. Zhou et al. propose archiving links and predicting potential link failure to mitigate and avoid “404 Not Found” errors [37]. 3.4 Query Rot Query Rot The issue of query performance and stability over time (see Definition 4), has been indirectly addressed in various studies on SPARQL querying such as Verborgh et al. “Querying Datasets on the Web with High Availability”[30], but has not been explicitly named or systematically studied before our work. 4 Mitigating Query Rot using snapquery The first step in mitigating Query Rot is to proactively identify when a query is no longer working as expected. Snapquery makes query sets available in computer readable form and has a command line and web interface to execute a query set against fitting endpoints and knowledge graphs. A whole Query Set of a project / Query Execution Context may be analyzed automatically this way, e.g., in a Continuous Integration (CI) and Continuous Delivery (CD) pipeline. Figure 1 shows an example analysis. A difference in the number of failures and successes for a set of endpoints indicates potential Query Rot. In the example, the endpoints “Wikidata” and “Wikidata-scatter” should behave 100% identically but they do not. Wikidata-qlever performing differently on the examples and named-queries namespaces is to be expected since it does not support Blazegraph special syntax and is not feature complete yet. Fig. 1: Query Set Success Overview 16 Wolfgang Fahl, Christoph Lange, Tim Holzheim, and Stefan Decker References 1. AD Freiburg: Qlever: The efficient query engine. https://github.com/ ad-freiburg/qlever (2024), https://github.com/ad-freiburg/qlever, gitHub repository 2. Alexiev, V.: Query Parameterization ·Issue #57 ·w3c/sparql-dev — github.com. https://github.com/w3c/sparql-dev/issues/57 (2019), [Accessed 12-07-2024] 3. Bast, H., Buchhold, B.: Qlever: A query engine for efficient sparql+text search. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. CIKM ’17, ACM (Nov 2017). https://doi.org/10.1145/3132847.3132921, http://dx.doi.org/10.1145/3132847.3132921 4. Coble, Z., Karlin, J.: Reference rot in the digital humanities literature: An analysis of citations containing website links in DHQ. Digit. Humanit. Q. 17(1) (2023), http://www.digitalhumanities.org/dhq/vol/17/1/000662/000662.html 5. Cyganiak, R., Wood, D., Lanthaler, M.: RDF 1.1 Concepts and Abstract Syntax. https://www.w3.org/TR/rdf11-concepts/ (2014), accessed: 2024-07-10] 6. Fahl, W.: T363894 Introduce Named Queries and Named Query Middleware to wikidata — phabricator.wikimedia.org. https://phabricator.wikimedia.org/ T363894 (2024), [Accessed 12-07-2024] 7. Fahl, W.: Workdocumentation 2024-05-12 - research wiki — cr.bitplan.com. https://cr.bitplan.com/index.php/Workdocumentation_2024-05-12#LLM_ Query_name_annotation (2024), [Accessed 12-07-2024] 8. Fahl, W., Holzheim, T.: Category:Snapquery - research Wiki — cr.bitplan.com. https://cr.bitplan.com/index.php/Category:Snapquery (2024), [Accessed 1207-2024] 9. Fahl, W., Holzheim, T.: Snapquery Cats LLM Rewrite Experiment - research wiki — cr.bitplan.com. https://cr.bitplan.com/index.php/Snapquery_Cats_ LLM_Rewrite_Experiment (2024), [Accessed 12-07-2024] 10. Fahl, W., Holzheim, T., Lange, C., Decker, S.: Semantification of ceur-ws with wikidata as a target knowledge graph. In: Joint Proceedings of TEXT2KG 2023 and BiKE 2023. CEUR Workshop Proceedings (2023), https://ceur-ws.org/ Vol-3447/Text2KG_Paper_13.pdf 11. Fahl, W., Holzheim, T., Lange, C., Decker, S.: Sempubflow: A novel scientific publishing workflow using knowledge graphs, wikidata, and llms – the ceur-ws use case (2024), manuscript submitted for publication 12. Fahl, W., Holzheim, T., Westerinen, A., Lange, C., Decker, S.: Getting and hosting your own copy of wikidata. In: 3rd Wikidata Workshop 2022. Proceedings. Fraunhofer-Gesellschaft (2022). https://doi.org/10.24406/PUBLICA-976, https: //publica.fraunhofer.de/handle/publica/437118 13. Fahl, W., Holzheim, T., Westerinen, A., Lange, C., Decker, S.: Property cardinality analysis to extract truly tabular query results from wikidata. In: Kaffee, L.A., Razniewski, S., Amaral, G., Alghamdi, K.S. (eds.) Proceedings of the 3rd Wikidata Workshop 2022, co-located with the 21st International Semantic Web Conference (ISWC2022). CEUR Workshop Proceedings, vol. 3262. Wikidata Workshop 2022, CEUR Workshop Proceedings (Oct 2022), https://ceur-ws.org/Vol-3262/ paper7.pdf 14. Fahl, W., Holztheim, T., Priskorn, D.: GitHub - WolfgangFahl/snapquery: Frontend to Introduce Named Queries and Named Query Middleware to wikidata — github.com. https://github.com/WolfgangFahl/snapquery (2024), [Accessed 1207-2024] Mitigating Query Rot with snapquery 17 15. Feigenbaum, L., Williams, G.T., et al.: Sparql 1.1 protocol. Tech. rep., W3C (2013), https://www.w3.org/TR/2013/REC-sparql11-protocol-20130321/ 16. Harris, S., Seaborne, A., et al.: Sparql 1.1 query language. Tech. rep., W3C (2013), https://www.w3.org/TR/sparql11-query/ 17. Hennessey, J., Ge, S.X.: A cross disciplinary study of link decay and the effectiveness of mitigation techniques. BMC Bioinformatics 14 (2013). https://doi.org/10.1186/1471-2105-14-s14-s5 18. Lange, C., Di Iorio, A.: Semantic publishing challenge – assessing the quality of scientific output. In: Communications in Computer and Information Science, pp. 61–76. Springer International Publishing (2014). https://doi.org/10.1007/978-3319-12024-9_8, https://doi.org/10.1007/978-3-319-12024-9_8 19. Lorey, J.: Sparql endpoint metrics for quality-aware linked data consumption. In: Proceedings of International Conference on Information Integration and Web-based Applications & Services. IIWAS ’13, ACM (Dec 2013). https://doi.org/10.1145/2539150.2539240, http://dx.doi.org/10.1145/ 2539150.2539240 20. Lorey, J.: What’s in a Query: Analyzing, Predicting, and Managing Linked Data Access. Dissertation zur erlangung des akademischen grades doktor der ingenieurwissenschaften (dr.-ing.), University of Potsdam, Potsdam, Germany (Mar 2014) 21. Lorey, J., Naumann, F.: Detecting SPARQL Query Templates for Data Prefetching, p. 124–139. Springer Berlin Heidelberg (2013). https://doi.org/10.1007/978-3-64238288-8_9, http://dx.doi.org/10.1007/978-3-642-38288-8_9 22. Martin, R.C.: Design principles and design patterns (2000), https: //web.archive.org/web/20150906155800/http://www.objectmentor.com/ resources/articles/Principles_and_Patterns.pdf, originally published on objectmentor.com. Archived by the Wayback Machine on September 6, 2015 23. van Merode, H.: Continuous Integration (CI) and Continuous Delivery (CD): A Practical Guide to Designing and Developing Pipelines. Apress (2023). https://doi.org/10.1007/978-1-4842-9228-0, http://dx.doi.org/10.1007/ 978-1-4842-9228-0 24. Nielsen, F.Å., Mietchen, D., Willighagen, E.: Scholia, scientometrics and wikidata. In: The Semantic Web: ESWC 2017 Satellite Events. p. 237–259. Springer International Publishing (2017). https://doi.org/10.1007/978-3-319-70407-4_36, http://dx.doi.org/10.1007/978-3-319-70407-4_36 25. Oracle: Java naming conventions. https://www.oracle.com/java/technologies/ javase/codeconventions-namingconventions.html (1999), [Accessed 13-072024] 26. Parnas, D.L.: On the criteria to be used in decomposing systems into modules. Communications of the ACM 15(12), 1053–1058 (Dec 1972). https://doi.org/10.1145/361598.361623, http://dx.doi.org/10.1145/361598. 361623 27. Pérez, J., Arenas, M., Gutierrez, C.: Semantics and complexity of sparql. ACM Transactions on Database Systems (TODS) 34(3), 1–45 (2009) 28. Prud’hommeaux, E., Seaborne, A., et al.: SPARQL Query Language for RDF — w3.org. W3c recommendation, W3C (2008), https://www.w3.org/TR/ rdf-sparql-query/, [Accessed 2024-06-26] 29. Tyler, D.C., McNeil, B.: Librarians and link rot: A comparative analysis with some methodological considerations. portal: Libraries and the Academy 3, 615– 632 (2003). https://doi.org/10.1353/pla.2003.0098 18 Wolfgang Fahl, Christoph Lange, Tim Holzheim, and Stefan Decker 30. Verborgh, R., Hartig, O., Meester, B.D., Haesendonck, G., Vocht, L.D., Sande, M.V., Cyganiak, R., Colpaert, P., Mannens, E., de Walle, R.V.: Querying datasets on the web with high availability. In: Lecture Notes in Computer Science, International Semantic Web Conference. pp. 180–196. Springer, Springer International Publishing (2014). https://doi.org/10.1007/978-3-319-11964-9_12, http: //dx.doi.org/10.1007/978-3-319-11964-9_12 31. Warren, P.: Knowledge management and the semantic web: From scenario to technology. IEEE Intelligent Systems 21(1), 53–59 (Jan 2006). https://doi.org/10.1109/mis.2006.12, http://dx.doi.org/10.1109/MIS.2006.12 32. Warren, P., Alsmeyer, D.: The digital library: a case study in intelligent content management. Journal of Knowledge Management 9(5), 28–39 (Oct 2005). https://doi.org/10.1108/13673270510622438, http://dx.doi.org/10. 1108/13673270510622438 33. Warren, P., Mulholland, P.: Using SPARQL – The Practitioners’ Viewpoint, p. 485–500. Springer International Publishing (2018). https://doi.org/10.1007/978-3030-03667-6_31, http://dx.doi.org/10.1007/978-3-030-03667-6_31 34. Warren, P., Mulholland, P.: A Comparison of the Cognitive Difficulties Posed by SPARQL Query Constructs, p. 3–19. Springer International Publishing (2020). https://doi.org/10.1007/978-3-030-61244-3_1, http://dx.doi.org/ 10.1007/978-3-030-61244-3_1 35. Wilkinson, M.D., Dumontier, M., Aalbersberg, I.J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.W., da Silva Santos, L.B., Bourne, P.E., Bouwman, J., Brookes, A.J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C.T., Finkers, R., Gonzalez-Beltran, A., Gray, A.J., Groth, P., Goble, C., Grethe, J.S., Heringa, J., ’t Hoen, P.A., Hooft, R., Kuhn, T., Kok, R., Kok, J., Lusher, S.J., Martone, M.E., Mons, A., Packer, A.L., Persson, B., Rocca-Serra, P., Roos, M., van Schaik, R., Sansone, S.A., Schultes, E., Sengstag, T., Slater, T., Strawn, G., Swertz, M.A., Thompson, M., van der Lei, J., van Mulligen, E., Velterop, J., Waagmeester, A., Wittenburg, P., Wolstencroft, K., Zhao, J., Mons, B.: The FAIR guiding principles for scientific data management and stewardship. Scientific Data 3(1) (mar 2016). https://doi.org/10.1038/sdata.2016.18, https://doi.org/10.1038/data.2016.18 36. Williams, H.K.R., Elder, R.: Introducing the wikidata thesis toolkit. Paper presented at the CILIP Metadata and Discovery Group Conference: "ReDiscovery", IET Birmingham: Austin Court, Birmingham, United Kingdom (2023), http://eprints.lse.ac.uk/120224/, accessed: 2023-07-13 37. Zhou, K., Grover, C., Klein, M., Tobin, R.: No more 404s: Predicting referenced link rot in scholarly articles for pro-active archiving. In: Proceedings of the 15th ACM/IEEE-CS Joint Conference on Digital Libraries. ACM (2015). https://doi.org/10.1145/2756406.2756940