Material SciKGDash
Abstract
This supplementary material accompanies the paper "SciKGDash: The Scientific Knowledge Graph Dashboard for Supporting Knowledge Curation." The file User_Stories.csv/User_Stories.xlsx contains the initial user stories defined by the curation and community-building (C&CB) team of the Open Research Knowledge Graph (ORKG). These stories represent the requirements that motivated the development of SciKGDash, including their assigned priorities, implementation status, and mapping to the team’s needs. To evaluate the dashboard, we conducted a design science experiment. The file Experiment_instructions.pdf provides the participant materials, including demographic questions, task descriptions with contextual explanations, and assessment instruments. These instruments cover task-specific scales (after each task), as well as SUS scores and need-fulfillment ratings at the end. The file Experiment_results.csv reports the collected results. It contains both objective and subjective measurements, covering effectiveness (task completion), efficiency (time taken), and subjective satisfaction based on short UEQ values.
Full text
Experiment for evaluating ”SciKGDash” Thank you for you participation! Please first fill in the fields for demographic information. The data will be anonymized. Name: Age: Job description: Working at ORKG since: No answer No answer No answer To which team do you belong? Curation & Community Building Team General ORKG Team External In how many of the previous evaluations of the dashboard with the Curation & Community Building Team were you present? I understand, that my data is only stored for statistical purposes and is processed anonymously. I hereby consent, that the screen and sound is recorded while i participate. The data is not published and only necessary for analytic purposes. I understand, that I can always withdraw my participation and my data gets deleted. I confirm, that I voluntarily take part in the experiment. Date, Signature: You have 4 tasks in total for the sections of the dashboard. Please take your time and read the instructions carefully. Please feel encouraged to express your thoughts at any time and don’t hesitate to ask questions. Your input and feedback are highly valued throughout the experiment.
1 Overview Navigate to the Overview tab. Duplicate Predicates You are in the role of a curator. Your task is to merge redundant Predicates as this will improve the overall quality of the knowledge graph. Start with merging Predicates with the fewest duplicates. To avoid merging Predicates that have the same label but describe different concepts, your focus will be on Predicates that lack a description. 1. Identify the Predicate with the least amount of duplicates which is lacking a description. 2. Open the respective pages in the ORKG. Strongly disagree Disagree Neutral Agree Strongly agree I finished the task fast. I finished the task successfully. 2 Visitors Navigate to the Visitors tab. You are in the role of a community building expert. You specialized in user behaviour analysis and your tasks involve studying visitor data within the ORKG. Focusing on a specific time span (March 6, 2024 to March 7, 2024), you aim to identify the most traversed paths and areas of interest among visitors. By evaluating this observational data, you can provide suggestions to improve the overall structure of the ORKG for optimal user experience and community engagement. Network graph 1. You want to see visitor statistics from March 6, 2024 to March 7, 2024. Set the respective dates and fetch the data. 2. Modify the graph by changing the parameters on the left side according to your approval for best visualization. 3. Identify the most frequently used edge in the network. 4. For the target node of this edge: Identify to which node most of the visitors would traverse after this. Strongly disagree Disagree Neutral Agree Strongly agree I finished the task fast. I finished the task successfully. Frequent Paths Now you want to enhance your understanding of visitor navigation patterns. With a focus on longer paths, your aim is to identify the most frequently used route within the specified time span. 1. Identify the most used path of minimum length 3. Determine the number of occurences. Strongly disagree Disagree Neutral Agree Strongly agree I finished the task fast. I finished the task successfully. Page 2
3 Insights Navigate to the Insights tab. Statements per Paper You are in the role of a curator. Your job is to keep track of the quality and comprehensiveness of the ORKG’s content. To ensure that all Papers meet the established standards of completeness, your task is to identify Papers with minimal Statements. 1. Identify the Paper with the least amount of Statements and open it in the ORKG. Strongly disagree Disagree Neutral Agree Strongly agree I finished the task fast. I finished the task successfully. 4 Actions Navigate to the Actions tab. You are in the role of a curator. You are responsible for maintaining the quality and relevance of resources. By pointing at negative aspects of improvement as well as positive examples of modeling, this practice maintains principles of transparency and responsiblity. Your goal is to provide thoughtful feedback on a specific resource. 1. Make a comment for the Paper with the least amount of Statements. Choose the type of comment which seems applicable to you and add a description of you choice. Strongly disagree Disagree Neutral Agree Strongly agree I finished the task fast. I finished the task successfully. Thank you for taking your time with the tasks! I would greatly appreciate it if you could take a moment to provide some concluding thoughts and feedback. Page 3
User Experience Questionnaire Please fill in the following table to rate your experience with the dashboard. In my opinion, the dashboard is ... obstructive supportive complicated easy inefficient efficient confusing clear boring exciting not interesting interesting conventional inventive usual leading edge Adressed needs Please fill in the following table to rate if the needs of the Curation & Community Building Team are adressed by the dashboard. Strongly disagree Disagree Neutral Agree Strongly agree The dashboard allows the tracking of visitor paths. The dashboard makes it possible to indicate if a resource is inaccurate or incomplete. The dashboard provides tools for monitoring the quality of the ORKG. The dashboard provides an overview of areas for improvement. The dashboard lists how active the internal (ORKG members) and external users are. You have now the opportunity to browse through the dashboard if you like to and give me some feedback. Feedback Do you have any feedback? What are suggestions for improvement? Your input is highly appreciated. Page 4