Service Evaluation Guide

Service Evaluation Guide Service Evaluation Guide Susanne Arndt, Tilmann Wittenhorst, Anette Ganske, Dorian Lehmenkühler Version 1, July 2025 Purpose and target group of this document The purpose of this document is to guide service teams in the Base4NFDI Initialisation Phase through their software and service evaluation, which is a prerequisite for their own basic service development. The document gives recommendations on conducting such an evaluation. It supports service teams in creating their deliverable D.TA1.2 Evaluation of Existing Software and Services.1 Reminder - Deliverable D.TA1.2 The evaluation and its documentation (summary of findings) is recommended to be completed within the first six months of the Initialisation Phase if you wish to apply for the Integration Phase. The service evaluation must be documented in the respective task in the Open Project of the service team. Alternatively, the evaluation results can be a separate, open document, which is preferably published (e.g. at Zenodo). In this case, a link to the document/ publication must be provided in Open Project, optionally (but preferably) accompanied by a short summary of this deliverable . Evaluation of Tools, Methods, Frameworks and Technologies Scope and Foundation of the Evaluation The goal of the evaluation is about exploring suitable service components. The evaluation should therefore encompass all tools, methods, frameworks and technologies that are potentially capable of satisfying a substantial part of the identified user requirements the basic service is supposed to address. It should therefore include all items that have the potential to be a solution for the user needs or to become a major component within the design of a basic service. If you already evaluated components in your proposal, use this as a starting point: Reference the proposal in deliverable D.TA1.2. Review whether your evaluation considered all of the points addressed below. Concentrate on new and missing aspects in D.TA1.2 and update, extend or supplement proposal contents where necessary. 1 The TA1 team can also support this process by monitoring the service landscape and ensuring that there are no blind spots. Service Stewards can assist you, in particular by identifying current technology trends within the consortia and their user bases. Base4NFDI Service Evaluation Guide 1 Service Evaluation Guide The evaluation of tools, methods, frameworks and technologies can only start after a requirements analysis has been performed (deliverable D.TA1.12). You should be aware of the different personas identified for the service, their main needs as well as the main problem areas for use cases. This input allows you to derive all the prospective features of your basic service which will then serve as the evaluation criteria, against which candidates can be evaluated. For detailed, step-by-step instructions on how to perform a software evaluation, consult the Appendix: From User Requirements to Software Candidates. Landscape Analysis A comprehensive landscape analysis of research data management software and services is indispensable for strategically directing your service development toward genuine needs and untapped potential. The aim of the evaluation is to identify solutions for a stable, secure, sustainable and sufficiently scalable basic service. It also helps you to early on identify and familiarize yourself with upcoming initiatives that may become relevant for your service in the future (e.g. EOSC). You should basically answer the question: How does your proposed basic service integrate itself into the landscape, i.e. ● with respect to similar/ competing services (nationally and internationally), ● with respect to services that shall use your basic service? If you have not done so already, try to identify which services or software products exist that are similar to your proposed basic service. Give a brief, structured overview over these services (e.g. developer/ provider, title, source code, helpdesk/ issue tracker, reference release). ● Are there any new services since you wrote the proposal? ● Should you evaluate these new services/ products as potential candidates for the development of your own service? Why not? ● Which of the services you identified are competing with your service proposal? How likely is it that competitors will provide the functionalities/ service offerings by the end of your Initialisation Phase? ● Does your service still address the gap identified by your proposal? Is there still added value resulting from your basic service development? Should the development plan be adjusted? You should especially observe developments within German National Research Data Infrastructure (Nationale Forschungsdateninfrastruktur (NFDI)), the European Open Science Cloud (EOSC)3, but also other potential solutions and components on the market. Try to describe the positioning of your service within the NFDI as a whole. Please specify to what extent the planned basic service overlaps with services provided by the EOSC EU Node, whether it would be complementary, competing, or an entirely new offering. 3 For EOSC services refer to https://open-science-cloud.ec.europa.eu/services (last accessed: July 2, 2025) 2 Note that if you wish to apply for the Integration Phase, deliverable D.TA1.1 should be completed approximately 3 months after the official start of the Initialisation Phase Base4NFDI Service Evaluation Guide 2 Service Evaluation Guide Documenting your Reasons and Choices The decision for some software or service should be made transparent by your service evaluation report. You should document ● reasons for choosing a software/ a service, but also ● reasons for its rejection. In the Initialisation Phase, you should reconsider whether you have fully estimated which is the best candidate (cf. Appendix: From User Requirements to Software Candidates). Do not just gather a list of functionalities and service packages with a check mark indicating whether they are available in a software/ service or not! The report is supposed to make the judgement and the final choice transparent. ● Why is one particular software or service the best choice for your basic service? ● Why are softwares/ services not sufficient to be the foundation of your basic service? ● If several candidates are suitable in different contexts, try to provide information about these contexts and their conditions. It is recommended to respond to any comments regarding software and technology evaluation that were given by the Technical Expert Committee (TEC) during the review of your Initialisation Phase proposal. Optional Appendices You may add appendices to your service evaluation report, if these help to clarify the choice for one of your candidates. These could be ● results from user acceptance testing (UAT) or pilot programs, ● stakeholder feedback summaries, ● developer’s demonstration notes. Requirements and Recommendations from Base4NFDI Base4NFDI requires you to choose software that has an open and ideally permissive license4. Please also make sure that licenses of all components are compatible with each other5. For all components consider the technical, legal, operational, and semantic aspects that will affect the use of this component with your service and larger infrastructures like EOSC. Furthermore, Base4NFDI recommends that software and services ● follow a version control scheme (check release history), ● employ widely-used, ideally standardised schemata for (meta-)data modelling, ● have an active, international community, (check stars, forks, issues), 5 The LIcensing Assistant of the European Commission can also assist you with a compatibility assessment of a self-chosen set of licenses. https://interoperable-europe.ec.europa.eu/collection/eupl/solution/licensing-assistant/compatibi lity-checker (last accessed: June 24, 2025) 4 The Licensing Assistant of the European Commission can help you to find an appropriate license. We recommend setting the filters ‘permissive’ and ‘strong community’. https://interoperable-europe.ec.europa.eu/collection/eupl/solution/licensing-assistant/find-andcompare-software-licenses (last accessed: June 4, 2025) Base4NFDI Service Evaluation Guide 3 Service Evaluation Guide ● engage in communication with their community (look for forums, Slack/Discord, mailing lists, or active issues), ● are open to suggestions from their community, (check pull requests, activity) ● have at least technology-readiness level (TRL) 3–4 at the beginning of the Initialisation Phase and TRL 5–6 by the end of the Initialisation Phase, ● should have basic log capabilities and allow for monitoring and gathering certain metrics, ● do not depend on a single vendor company (avoid vendor lock-in), ● should be ready to follow a basic set of software quality-criteria which will become relevant for the Integration and Ramp-up Phase in Base4NFDI6, ● follow accessibility requirements to support all kinds of user groups7, ● follow basic requirements regarding software ergonomics8. Risk Assessment and Mitigation Strategies The software evaluation should also have a closer look at potential risks associated with software/ service candidates. We advise you to address foreseeable risks posed by your chosen candidate(s) by proposing mitigation strategies. Reference the SWOT-analysis and section II.4.4 Risks and Challenges (v. 3.3 03/2025) of your Initialisation Phase proposal if these have already discussed the candidate(s). You should make a good argument why and how your candidate(s) is/are still fit to be a sound foundation of a basic service for the NFDI – even if certain details must be left open for later stages of the development. In particular, you should provide strategies when your basic service relies on software or services that ● is commercial: how will the use of such services be financed? ● is unstable, e.g. due to early releases, frequent upstream changes: how do you guarantee stable operation of your service? ● is unreliably maintained, no longer supported, not well documented – how do you deal with complications? ● has unclear testing and quality assurance routines: how will you check and guarantee the codebase’s health and clean architecture? ● has foreseeable security issues: how has the provider responded to prior security issues? How severe would impacts from security issues be in the context of the basic service? ● has foreseeable scalability issues: is it possible to use cloud features and containerization? ● utilizes software that is not widespread in the target community – how do you facilitate the cultural change in your community? 8 cf. ISO/TR 9241-100:2023 Ergonomics of human-system interaction - Part 100: Overview of ISO 9241 software ergonomic standards. 7 Base4NFDI will provide an extensive guide on how to comply with legal accessibility requirements. The document will be available on Zenodo, where the DOI 10.5281/zenodo.15719661 was reserved. 6 cf. notes_on_software_quality.pdf, supplement of Schäfer-Neth, C., Grudskaia, A., Chen, T., & Giesler, A. (2025). Report on Base4NFDI service integration procedures (D2.1.1) (1.1). Zenodo. https://doi.org/10.5281/zenodo.15235863 and quality aspects mentioned in the integration phase proposal template on the Base4NFDI website: https://base4nfdi.de/resources/templates-logos (last accessed: June 4, 2025) Base4NFDI Service Evaluation Guide 4 Service Evaluation Guide Appendix: From User Requirements to Software Candidates This section of the guide provides step-by-step instructions for software evaluation. Its use is optional. Please consider using it, if you find the procedure helpful. Assessing User Requirements, Evolving Service Features ● Review the key findings from your (previously completed) requirements analysis (deliverable D.TA1.19): What are the main needs of the user groups (target groups) for your basic service? ● What could be the main problem areas (‘pain points’) when addressing your users’ use cases? (Your personas could also tell you more.) What could prevent users from getting benefit out of the basic service? Any blind spots that might have been overlooked? ● Identify all features a basic service has to deliver to fit into the use cases and to resolve the ‘pain points’. Remember: software evaluation is about why a basic service is needed and about exploring suitable software products - how to deliver a basic service is part of the next steps, the actual service design and development. ● Divide these features into required and preferred. The former define the evaluation criteria. Do not exclusively think about user-related functionality during the evaluation. There are further aspects you need to consider, like ○ integration with existing systems of the target group, ○ reporting & analytics functionality, ○ data security, support of data export and import, open formats, ○ UX aspects (e.g. efficient workflows), ○ setup and operation requirements/ complexity, ○ additional life-cycle costs (documentation, training, training materials, maintenance and updating, scalability, compliance aspects, content curation), ○ provider aspects (product roadmap, provider stability). Setting up an Evaluation Environment for a Software Service ● Research possible software for the basic service (if you haven’t already done so for the Initialisation Phase proposal). Make a shortlist of the main candidates. Only these need to be evaluated and their evaluation documented (deliverable D.TA1.2). Prefer software that is open and FAIR10 and maintained by scientific communities or infrastructure providers. ● Estimate the scope of the evaluation: How many features are required? How many software candidates are available? Are they all openly available/ downloadable? Is a review of the documentation sufficient, or is a trial installation required (look out for ready-to-use test platforms)? 10 See e.g. Barker, M., et al.: Introducing the FAIR Principles for research software. Sci Data 9, 622 (2022). https://doi.org/10.1038/s41597-022-01710-x 9 Note that if you wish to apply for the Integration Phase, deliverable D.TA1.1 should be completed approximately 6 months after the official start of the Initialisation Phase Base4NFDI Service Evaluation Guide 5 Service Evaluation Guide ● Map the features of the software candidates and the required features into evaluation criteria. The evaluation criteria break the requirements down into specific, ideally quantifiable or otherwise objectively verifiable subcriteria that enable you to evaluate the software candidates11. ● If you are evaluating a small number of software products and/or features (approximately less than 5 each): Take a practical approach rather than following the rules - but remain neutral! If you already have a preference, it is better to opt for a comprehensible evaluation. If you are evaluating a larger number of software products and/or features, please perform a weighted utility analysis (see Table 1 below). Getting the Big Picture and Choosing a Winner ● Is there a clear, distinct winner? Perhaps several software products are on equal footing and further evaluation (with additional criteria) is recommended. ● Regarding the winner: are there any outstanding issues or is the evaluation result questionable? Seek assistance from the software vendor/maintainer or service stewards, or TA1, to resolve outstanding issues. ● For the winner, is there a commitment to continuous improvement of the software? Is it well maintained and released under an open source licence? ● Draw a conclusion: Is anything from the requirements analysis a showstopper for the highest scoring software? Will the stakeholders accept the choice? Will the users accept the product? Will IT administrators and IT security staff be able to maintain it? ● Share the results with the stakeholders and other decision makers (possibly to get a second opinion) and write the results into the report for the deliverable D.TA1.2. Table 1: Weighted utility analysis Excursion: Weighted Utility Analysis For a weighted utility analysis perform the following steps: ● List all applicable features and weigh them by percentage for each software product individually (i.e. the sum being 100 over all features, per software product) ● Based on your research/testing, assign a rating from 1 (poor) to 5 (excellent) for each feature per software product. ● Calculate the score for each feature per software product by the formula ( [weighted feature percentage] ÷ 5 ) × rating = score ● Add up all the scores per software product to get the final score. 11 An example would be “collaborative document editing in a cloud service” – evaluation criteria for such a feature could be a) feature is present (yes/no), b) there is no perceivable lag on different user systems when editing a document together, c) there is no data loss in between user systems and the remote environment. Base4NFDI Service Evaluation Guide 6