Questionnaire for Evaluating a Metadata Schema for Energy Research Software

1 Questionnaire for Evaluating a Metadata Schema for Energy Research Software Stephan Ferenz 1,2 https://orcid.org/0000-0001-9523-7227, Oliver Werth 1 https://orcid.org/0000-0002-6767-5905 1 OFFIS e.V. – Institute for Information Technology, Oldenburg, Germany 2 Carl von Ossietzky University of Oldenburg, Oldenburg, Germany This questionnaire was used to evaluated a metadata schema for energy research software (ERSmeta: https://github.com/NFDI4Energy/ERSmeta) and a tool for semi-automated metadata creation (SMECS: https://github.com/NFDI4Energy/SMECS). Introductory Part Title: Metadata for Energy Research Software Introduction Dear survey participant, as part of NFDI4Energy, we are conducting a survey on "metadata for energy research software". By means of the survey, we aim to evaluate our metadata schema for energy research software which enables energy researchers to create structured metadata for their research software. Energy research software is defined as software used in the scientific discovery process for understanding, analyzing, improving, and/or designing energy systems or components of the energy system. Metadata is data about the software which allow others to better find, understand, and reuse the software. This survey will take approximately 60 minutes and is divided into four parts: 1. Classification of your background experience (approx. 5 min.) 2. Experiment: Create metadata for your energy research software (approx. 35 minutes) 3. Evaluation of usability (approx. 15 minutes) 4. Voluntary upload of created metadata (approx. 5 minutes) For further information about our research or the survey, please contact us at [email protected]. The survey will remain open until August, 15th. Voluntariness and Anonymity Participation in the study is voluntary. You may withdraw from the study at any time and without providing reasons, without facing any disadvantages. The data and personal information collected during this study, as described above, will be treated confidentially. Project staff who have direct contact with you and access to personal data are bound by confidentiality obligations. Furthermore, the study results will be published in an anonymized form, meaning that your data cannot be linked to you personally. After creating the metadata in the second step of the survey, you will have the option to submit the metadata for further research purpose (e.g., showing the metadata within further planned evaluation). The submission of the metadata is completely voluntary and is not required to finalize this survey. 2 Data Protection Data collection is carried out in full anonymization, meaning that your name or other identifying personal data will not be requested or stored at any point. Compensation Participation in this study will not be compensated. Best regards, Stephan Ferenz Questions Related Literature 1. General Information on Your Research Background 1. Which institution are you affiliated with? (Multiple choice) o University o Non-University Research Institution o University of Applied Sciences o Other Scientific Institution o Industry 2. What is your position within your institution? (Single choice) o PhD Student o Postdoctoral Researcher o Junior Professor o Professor o Other Research Staff o Research-Supporting Staff o Retired o Other (Please specify) 3. How many years have you worked in research? (Single choice) o <4 years o 5-9 years o 10-19 years o 20 years and more 4. To which research discipline do you primarily belong? (Single choice) o Mechanical, Electrical or other Engineering o Computer Science o Natural Science o Social Science o Economics 5. Please describe your research topics in three keywords. (Open text) 2. Experience with Metadata and Energy Research Software 1. How familiar are you with metadata schemas? (Single choice) o I have participated in the development of a metadata schema. o I have used a metadata schema to create metadata. 3 o I know what a metadata schema is but have not applied one yet. o I am not familiar with metadata schemas. o I have never heard about metadata schemas. 2. Which experience on energy research software did you already made in the past? (Multiple choice) o I have been involved in the development of one or more energy research software. o I have used an energy research software for my research. o I know what an energy research software is but have not used one yet. o I am not familiar with energy research software. 3. How involved are you currently in the development of energy research software? (Single choice) o I am developing one energy research software. o I am developing multiple energy research software. o In my research group (if applicable), energy research software is currently developed without me. 4. Please choose one energy research software in whose development you are involved and for which you would like to create metadata in the next step. Please select the sentence which fits to that software. (Single choice) 1. The chosen software is openly available as a GitHub or GitLab repository. 2. The chosen software is in a closed GitHub or GitLab repository. 3. The chosen software is not available on GitHub or GitLab. 4. I cannot choose an energy research software. If software is available in open repository (1) → forward to next questions (01_Questionnaire_Middle) If the participant is not from research or does not have a software in a GitHub or GitLab repository at hand (2, 3, or 4) → show the following message (First completion Note): First Completion Note Dear Participant, Thank you very much for your interest in our survey. We truly appreciate your willingness to take part in our research. We regret to inform you that, unfortunately, you do not meet the necessary criteria to complete this evaluation. We were looking for researchers who have at least one energy research software in an open GitHub or GitLab repository at hand. Thank you once again for your time and interest. For further information about our research or the survey, please contact us at [email protected]. Best regards, Stephan Ferenz 4 Questionnaire: Middle Part Metadata Creation Dear survey participant, now, it is time for creating metadata for your energy research software. In the following, please go to SMECS using the login data below. In SMECS, please enter the link to a GitHub or GitLab repository of your energy research software to create metadata for the software in the following. Please proceed through SMECS and download the metadata (“Download JSON”) on the last page of the tool and save it locally where you can find it easily (There will be the option to voluntary upload the metadata at the end of this survey). After downloading and saving the metadata locally please come back to this questionnaire to continue the survey. URL: **** Username: **** Password: **** 1. Did you finalize the creation of metadata and download it to your machine? (Single choice) o Yes. o No, there were technical issues. o No, I do not have enough time for that. Yes. → The participant will go to the next questionnaire (02_Questionnaire_After). No, there were technical issues. → the participant will get the technical-failure message. No, I do not have enough time for that. → The participant will get the time message. Completion Note (Technical-failure message) Dear Participant, We are sorry to hear that you faced technical issues. Please write us a short mail with small description (and a screenshot) to [email protected] so we can fix the issue. Thank you very much for your interest in our survey. We truly appreciate your willingness to take part in our research. Thank you once again for your time and interest. For further information about our research or the survey, please contact us at [email protected]. Best regards, Stephan Ferenz Completion Note (Time message) Dear Participant, Thank you very much for your interest in our survey. We truly appreciate your willingness to take part in our research. Thank you once again for your time and interest. For further information about our research or the survey, please contact us [email protected]. Best regards, Stephan Ferenz 5 Questionnaire: After the Usage of SMECS Title: Metadata for Energy Research Software Introduction As last part of this survey, it is essential to assess your experience when creating metadata for your research software. Please take a few minutes to complete the following questions to provide your valuable feedback. Questions Related Literature 1. Evaluation of the tool First, the focus is on the tool (SMECS) which allowed the creation of the metadata. 1. System Usability Scale (SUS) For each of the following statements, please select the option that best reflects your experience with the system. Scale: Strongly Disagree | Disagree | Neutral | Agree | Strongly Agree 1. I think that I would like to use this system frequently. 2. I found the system unnecessarily complex. 3. I thought the system was easy to use. 4. I think that I would need the support of a technical person to be able to use this system. 5. I found the various functions in this system were well integrated. 6. I thought there was too much inconsistency in this system. 7. I would imagine that most people would learn to use this system very quickly. 8. I found the system very cumbersome to use. 9. I felt very confident using the system. 10. I needed to learn a lot of things before I could get going with this system. [1] 2. Behavioral Intention to use SMECS For each of the following statements, please select the option that best reflects your experience with the system. Scale: Strongly Disagree | Disagree | Neutral | Agree | Strongly Agree 1. I intend to use SMECS for another energy research software. 2. I will recommend SMECS to my colleagues. 3. I have positive perceptions about SMECS. 1. Inspired by “I intend to use the system in the next <n> months.” (Behavioral intention – UTAUT [2]) 2. Inspired by “I will recommend Lecture capture system to my friends also.” (Behavioral intention – UTAUT3 [3]) 3. Inspired by “I have positive perceptions about Lecture capture system (Behavioral intention – UTAUT3 [3]) 6 3. Quality of Extraction For each of the following statements, please select the option that best reflects your experience with the system. Scale: Strongly Disagree | Disagree | Neutral | Agree | Strongly Agree 1. The quality of the extracted metadata was high. 2. I had no problem with the quality of the extracted metadata. 3. Using the metadata extraction enhances the effectiveness of creating metadata. 4. I find the system to be useful to extract metadata. 1. Inspired by “The quality of the output I get from the system is high” (Output quality – TAM2 [4]) 2. Inspired by “I have no problem with the quality of the system’s output” (Output quality – TAM2 [4]) 3. Inspired by “Using the system enhances my effectiveness in my job.” (Perceived usefulness – TAM2 [4]) 4. Inspired by “I find the system to be useful in my job.” (Perceived usefulness – TAM2 [4]) 4. Additional Features For each of the following statements, please select the option that best reflects your wishes for improvement of SMECS. Scale: Strongly Disagree | Disagree | Neutral | Agree | Strongly Agree 1. Extracting additional information for my software from provided links to literature (e.g., software paper or other related paper) would have been helpful for me. 2. Extracting additional information from package repositories (e.g., pypi) would have been helpful for me. 3. I would like to have an option to export the metadata directly to my repository. 4. I would like to have an option to directly publish the metadata in a software registry. 5. Comment Questions: 1. What did you like most about SMECS? (Open text) 2. What did you dislike most about SMECS? (Open text) 3. Please provide any additional comments or suggestions for improving SMECS. (Open text) 2. Evaluation of the metadata schema Second, the focus is on the metadata schema which is the foundation for the created metadata and defined which information were part of the metadata. The metadata schema consists of multiple elements, i.e., properties, which need to be filled in the metadata schema. For example, one element is the element “name”, i.e., the name of the software. 1. Amount of elements For each of the following statements, please select the option that best reflects your experience with the system. 7 Scale: Strongly Disagree | Disagree | Neutral | Agree | Strongly Agree 1. I find the overall number of elements useful to describe my research software. 2. The amount of elements is too high. 3. The categorization of elements into three categories is helpful. 4. The number of mandatory elements is useful for me. 5. The number of recommended elements is useful for me. 1. Inspired by “I would find the system useful in my job.” (Perceived Usefulness - TAM [5], UTAUT [2]) 2. Usefulness of elements For each of the following statements, please select the option that best reflects your experience with the system. Scale: Strongly Disagree | Disagree | Neutral | Agree | Strongly Agree 1. Using the metadata schema to describe my research software would enable me to enrich my software with metadata more quickly. 2. I find the metadata schema useful to describe my research software. 3. I find it easy to use the metadata schema to describe my research software in the way I want. 4. I find the metadata schema to be flexible to use it. 5. I missed certain elements in the metadata schema to describe my research software. 6. The quality of the created metadata is high. 7. I have no problem with the quality of the created metadata. 8. Assuming I have access to the metadata schema, I intend to use it to describe my software. 9. Given that I have access to the metadata schema, I predict that I would use it to describe my software. 1. Inspired by “Using the system in my job would enable me to accomplish tasks more quickly” (Perceived Usefulness - TAM [5]) 2. Inspired by “I would find the system useful in my job.” (Perceived Usefulness - TAM [5], UTAUT [2]) 3. Inspired by “I would find it easy to get the system to do what I want it to do.” (Perceived Ease of Use – TAM [5]) 4. Inspired by “I would find the system to be flexible to interact with.” (Perceived Ease of Use – TAM [5]) 5. 6. Inspired by “The quality of the output I get from the system is high” (output quality - TAM2 [4]) 7. Inspired by “I have no problem with the quality of the system’s output” (output quality - TAM2 [4]) 8. Inspired by “Assuming I had access to the system, I intend to use it.” (behavioral intention - TAM3 [6]) 9. Inspired by “Given that I had access to the system, I predict that I would use it.” (behavioral intention - TAM3 [6]) 3. Descriptions of the elements For each of the following statements, please select the option that best reflects your experience with the system. Scale: Strongly Disagree | Disagree | Neutral | Agree | Strongly Agree 8 1. I find the descriptions useful for creating the metadata. 2. Using the descriptions enables me to describe my research software more quickly. 3. I thought there was too much inconsistency in the metadata elements. 4. Given the descriptions, it is easy for me to create metadata for my research software. 5. I could create metadata for my energy research software … a. If there was no one around to tell me what to do as I go. b. If I could call someone for help if I got stuck. c. If I had a lot of time to complete the metadata. d. If I had just the built-in help facility for assistance 1. Inspired by “I would find the system useful in my job.” (Perceived Usefulness - TAM [5], UTAUT [2]) 2. Inspired by “Using the system in my job would enable me to accomplish tasks more quickly” (Perceived Usefulness – TAM [5]) 3. Inspired by “I thought there was too much inconsistency in this system.” (SUS Questionnaire [1]) 4. Inspired by “Given the resources, opportunities and knowledge it takes to use the system, it would be easy for me to use the system.” (Perceptions of External Control - TAM3 [6]) 5. Inspired by “I could complete a job or task using the system …” (Self-efficacy - UTAUT [2]) 3. Usefulness of Value Vocabularies For certain elements, you were able to choose from a list instead of providing free text. For each of the following statements, please select the option that best reflects your experience with these provided lists. Scale: Strongly Disagree | Disagree | Neutral | Agree | Strongly Agree 1. I find the provided lists to be flexible to use it to describe my research software. 2. I find it easy to use provided lists to describe my research software in the way I want. 3. Using the provided lists to describe my research software would enable me to enrich my software with metadata more quickly. 4. I find the provided lists useful to describe my research software. 1. Inspired by “I would find the system to be flexible to interact with.” (Perceived Ease of Use – TAM [5]) 2. Inspired by “I would find it easy to get the system to do what I want it to do.” (Perceived Ease of Use – TAM [5]) 3. Inspired by “Using the system in my job would enable me to accomplish tasks more quickly” (Perceived Usefulness – TAM [5]) 4. Inspired by “I would find the system useful in my job.” (Perceived Usefulness – TAM [5], UTAUT [2]) 4. Comment Questions a. Which certain elements do you miss in the metadata schema? (Open text) 9 b. Which additional information on your software would you have liked to provide? (Open text) c. What did you like most about the metadata schema? (Open text) d. What did you dislike most about the metadata schema? (Open text) e. Please provide any additional comments or suggestions for improving the metadata schema. (Open text) People are forwarded to a new survey (03_Questionnaire_Metadata_Upload). References [1] J. Brooke, “SUS: A quick and dirty usability scale,” in Usability Eval. Ind., 1st Edition., vol. 189, CRC Press, 1995. doi: 10.1201/9781498710411-35. [2] V. Venkatesh, M. G. Morris, G. B. Davis, and F. D. Davis, “User Acceptance of Information Technology: Toward a Unified View,” MIS Q., vol. 27, no. 3, pp. 425–478, 2003, doi: 10.2307/30036540. [3] M. S. Farooq et al., “Acceptance and use of lecture capture system (LCS) in executive business studies: Extending UTAUT2,” Interact. Technol. Smart Educ., vol. 14, no. 4, Art. no. 4, Nov. 2017, doi: 10.1108/ITSE-06-2016-0015. [4] V. Venkatesh and F. D. Davis, “A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies,” Manag. Sci., vol. 46, no. 2, Art. no. 2, Feb. 2000, doi: 10.1287/mnsc.46.2.186.11926. [5] F. D. Davis, “Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology,” MIS Q., vol. 13, no. 3, Art. no. 3, 1989, doi: 10.2307/249008. [6] V. Venkatesh and H. Bala, “Technology Acceptance Model 3 and a Research Agenda on Interventions,” Decis. Sci. - Decis. SCI, vol. 39, pp. 273–315, May 2008, doi: 10.1111/j.15405915.2008.00192.x.