Scholars’ Perceptions of Relevance in Bibliography-Based People Recommender System

Computer Supported Cooperative Work (CSCW) (2019) 28:357–389 https://doi.org/10.1007/s10606-019-09349-w © The Author(s) 2019 Scholars’ Perceptions of Relevance in Bibliography-Based People Recommender System Ekaterina Olshannikova1,* , Thomas Olsson1, Jukka Huhtam¨ aki1& Peng Yao1 1Tampere University, Tampere, Finland (E-mail: ekaterina.olshanniko[email protected]; E-mail: [email protected]; E-mail: [email protected]; E-mail: [email protected]) Abstract. Collaboration and social networking are increasingly important for academics, yet identifying relevant collaborators requires remarkable effort. While there are various networking services optimized for seeking similarities between the users, the scholarly motive of producing new knowledge calls for assistance in identifying people with complementary qualities. However, there is little empirical understanding of how academics perceive relevance, complementarity, and diversity of individuals in their profession and how these concepts can be optimally embedded in social matching systems. This paper aims to support the development of diversity-enhancing people recommender systems by exploring senior researchers’ perceptions of recommended other scholars at different levels on a similar–different continuum. To conduct the study, we built a recommender system based on topic modeling of scholars’ publications in the DBLP computer science bibliography. A study of 18 senior researchers comprised a controlled experiment and semi-structured interviewing, focusing on their subjective perceptions regarding relevance, similarity, and familiarity of the given recommendations, as well as participants’ readiness to interact with the recommended people. The study implies that the homophily bias (behavioral tendency to select similar others) is strong despite the recognized need for complementarity. While the experiment indicated consistent and significant differences between the perceived relevance of most similar vs. other levels, the interview results imply that the evaluation of the relevance of people recommendations is complex and multifaceted. Despite the inherent bias in selection, the participants could identify highly interesting collaboration opportunities on all levels of similarity. Key words: People recommender systems, Social matching applications, Expert search, Homophily, Diversity, Research collaboration, Social networking, Bibliography data analysis 1. Introduction In scholarly work, collaboration has become a normative form of knowledge production. Researchers across the social sciences broadly concur that collaboration is the best path to solving complex problems and achieving exceptional results (Frydlinger et al. 2013). Collaboration is promoted as a means of cultivating quality, enhanced resource utilization, and high impact (Hsiehchen et al. 2015). In 358 Ekaterina Olshannikova et al. science and patenting, a substantial shift toward collective work has been found across scientific disciplines and business domains (Wuchty et al. 2007;B ¨ orner et al. 2010). In academic research, collaboration takes place on a dyadic level between individuals, amongst research teams, as well as within international consortia. However, identifying new suitable candidates for academic collaboration requires high investment in social networking, and the disciplinary structures can prevent unexpected combinations of individuals. Following this trend of the increasing importance of collaboration, supporting social networking and encouraging new social encounters have become central design goals in the HCI & CSCW communities. Prior research on so-called social matching (Terveen and McDonald 2005) has particularly looked into people recommender systems (Tsai and Brusilovsky 2016; Guy and Pizzato 2016)and opportunistic matching applications (Mayer et al. 2015a; Mayer et al. 2016) that aim to enable identification of new relevant connections, some of them employing playful approaches and gamification (Paasovaara et al. 2016). There are also prototypes of people recommender systems that specifically aim to match scholars: for instance, expert finding systems (Vassileva et al. 2003; Beham et al. 2010), or event-based mobile applications like ‘Find & Connect’ developed by Chin et al. (2014) and experimented at the UbiComp 2011 conference. Considering the rich publication data available in online repositories, prior research has looked into bibliography analysis methods for recommender systems, e.g., DBLP1-based systems for researchers (Zaiane et al. 2007). However, the majority of professional matching systems tend to utilize a similarity-maximizing approach, providing recommendations of like-minded others with similar interests. In this regard, Yuan and Gay (2006) deliberate that homogeneity produces both positive and negative effects on interpersonal communication, community formation, and knowledge work – “homophily not only unifies, it also divides a network”. On the one hand, collaboration within a group of people with shared interests can contribute to a safe and trustworthy work environment, enabling cohesive team spirit and ease of communication. On the other hand, it has been found that researching and cooperating with diverse individuals is essential in tasks that aim to create new knowledge (Mollica et al. 2003). Prior work emphasizes that an insightful dialogue between diverse actors can build social capital (Burt 2017) by increasing awareness about external knowledge groups and bridging polarized intellectual communities towards abounding knowledge sharing and idea creation (Argote and Ophir 2002). While research on diversifying item recommendations (Adamopoulos and Tuzhilin 2015; Castells et al. 2015) is gaining interest, few attempts have been made to match people based on diversity (Rajagopal et al. 2017). Additionally, the evaluation of people recommender systems and matching applications is geared toward the assessment of algorithm effectiveness, with little focus on user perceptions. Although there is well-established research on usercentered evaluation of content recommender systems (Knijnenburg et al. 2012; Scholars’ Perceptions of Relevance... 359 Pu et al. 2011), the choice of potential collaborators is significantly different and, therefore, requires contextually operationalized evaluation metrics. Considering the diverse needs of scholars, it is essential to pay attention to subjective perceptions regarding the recommended people and carefully conceptualize factors such as perceived relevance and willingness to follow-up on the recommendations. To enable the gathering of such data, we developed a simple DBLP-based people recommender system that provides the user with recommendations of other scholars from three different levels of similarity regarding their publication history – low, moderate and high. With a user study that combines a controlled experiment and semi-structured interviewing, we address the following research questions: (RQ1) What level of measured similarity of publication history is preferred in recommendations of potential collaborators? (RQ2) What specific needs and expectations scholars have in regard to seeking professional collaboration? The findings reveal an intriguing mismatch between scholars’ intuitive behavior and deliberate intentions regarding potential academic collaboration. While the quantitative results demonstrate participants’ general preference to most similar recommended people, the interview data brings up a variety of scholars’ needs for connecting with cross-disciplinary and diverse people. Thus, the nature of the collaboration task might influence the perceived relevance of potential candidates, for example, regarding the complementarity of professional roles, skills, and expertise. The contribution of this work is two-fold: (i) providing empirical findings on subjective perceptions of people recommendation relevance in the context of potential academic partnering, and (ii) presenting the qualitative account of academics’ needs in collaboration and factors that might affect their decision in choosing partners. Furthermore, as a methodological contribution, we operationalize measures for subjective opinions on people recommendations in the context of professional academic collaboration. 2. Related work While various disciplines have studied scientific collaboration in different ways, a general consensus is that collaboration is imperative for effective knowledge production. Bozeman et al. (2013) note that research collaboration is often limited to the notion of co-authorship, and criticize the assumption that cooperation is undoubtedly resulting in a knowledge product (e.g., scientific paper). In this article, we approach scholarly partnering practices, which are beyond co-authorship, and do not require an explicit valuable outcome. In fact, Bozeman and Corley (2004) define key motives for scientific collaboration and propose that the selection of collaborators can be driven by: (i) work ethic attribution and schedule compliance; (ii) shared nationality; (iii) need to mentor junior researchers; (iv) administration request or high reputation; (v) preceding collaboration experience, its quality and personality chemistry; (vi) complementarity of skills. Considering this breadth, in this article we adopt the broad definition by Bozeman et al. 360 Ekaterina Olshannikova et al. (2013): “collaboration is a social process whereby human beings pool their human capital for the objective of producing knowledge.” To emphasize novelty of our contribution and the research gap we cover the following topics. First, we discuss research on supporting social interaction and collaboration in the general context of conferences and introduce works on bibliography-based recommender systems. Then, we deliberate on the concepts of similarity and diversity. Finally, we provide an overview of existing user-centered evaluation metrics in recommender systems. 2.1. Computational support for matching scholars Supporting experts finding is one of the crucial CSCW design goals to facilitate collaborative knowledge creation and dissemination (Ackerman and McDonald 1996). Over the last two decades, systems for supporting the conference experience have expanded from increasing people’s awareness of necessary information at the venue (e.g., schedule and contents) to facilitating social encounters, for example arranging meetings (Nishibe et al. 1998) and general enhancement of attendee interaction with the environment and other people (Dey et al. 1999). One reasonably common approach relates to location and proximity-based services for finding relevant connections (Kawakita et al. 2004; Cox et al. 2003). For instance, ‘Find & Connect’, a social networking mobile application developed by Chin et al. (2014) and experimented during the UbiComp 2011 conference, aims to provide the users with social recommendations based on physical proximity and similarity of interests. The results of the work reveal that users preferred acting on familiar recommended people or friends-of-friends, as well as those who have similar research interests. There are also generic, platform-like services designed to help the attendees meet new people with shared interests (Zenk et al. 2014). An example of such tool is ‘Confer’ (Zhang et al. 2016) which has been tested and deployed in several HCI conferences. ‘Conference Navigator’ is another example and has gradually acquired new features and functions (Farzan and Brusilovsky 2007; Wongchokprasitti et al. 2010; Parra et al. 2012). For instance, the most recent version (Brusilovsky et al. 2017)–‘Conference Navigator 3’ – is a community-based recommender system that by utilizing content-based and tag-based analysis methods provide the user with personalized suggestions about people and contents of a conference. The user can explore their research community through interactive social network visualization and connect with similar experts either before or during an event. Another recent research proposes so-called ‘Adaptive Conference Companion’ (Arens-Volland and Naudet 2016) – a mobile application that aims to deliver personalized guidance for attendees of academic events. In addition to utilizing conference data and explicit user input, authors enhance profiling and matchmaking mechanism by extracting bibliographic database (DBLP, GoogleScholar) Scholars’ Perceptions of Relevance... 361 and social media channels (LinkedIn, ResearchGate and MyScienceWork). They applied a term frequency-inverse document frequency (TF-IDF)2algorithm (Beel et al. 2016) for recommending most similar people and sessions in the scope of users’ interests. The authors speculate that the majority of participants in the experiment were well-prepared for the conference and already had their schedule which matched with recommendation predictions of relevant content. Unfortunately, the authors did not discuss the effectiveness of a system from the social networking perspective, i.e., how people reacted to and perceived the relevance of people recommendations. Another vein of research relates to recommending academic collaborations by utilizing bibliography data and social networks analysis and, thus, suggesting candidates with similar research interests. For instance, Kong et al. (2016) utilized topic clustering model to retrieve academic domains, calculate authors’ features, and analyze academic collaboration networks. Another group of researchers (Li et al. 2014) explored co-authorship networks to identify relevant collaborators with an already existing academic tie. They observed that scholars’ relationships are more complicated in a real-world setting and suggested that future work should go beyond existing co-authorship networks and consider matching people without established connections. Most recently, Hoang et al. (2017) proposes a new approach to calculate similarity with deep learning and experiment on DBLP and WiKiCFP databases. Sie et al. (2012) designed a system for recommending future co-authors that utilizes co-authorship network and topic similarity aspects in the matching mechanisms. They tested researchers preferences regarding existing co-authors vs. new potential candidates with a light-weight user study by asking the participants to rate each recommendation on a scale from one to ten. The findings revealed that participants prefer existing connections with whom collaboration has been already established. To summarize, the prior work displays a diversity of research and development of people recommender services for scholars with a particular focus on analyzing human-generated content and publishing history. The literature on algorithmic approaches indicates that bibliographical data sets can serve as a valid data source for identifying and recommending social connections. The algorithmic choices encouraged us to approach the area with specific topic modeling methods (TF-IDF) and to analyze the cosine distance3(Li and Han 2013) between the authors. Although there is apparent interest in creating services for academic partnering, the primary contribution of preceding research lies in the design of new matching mechanisms and algorithms. The evaluation of such systems, thus, is focused on testing the quality criterion of prediction accuracy and little attention is paid to the subjective perceptions of recommendation usefulness and user’s intention to follow-up on those. In this article, we experiment on content-based similarity-difference dimensions and specifically focus on the human-centered and subjective evaluation of perceived relevance and related variables. 362 Ekaterina Olshannikova et al. 2.2. Concepts of similarity and diversity Our approach of diversity-enhancing people recommender systems is founded on the relatively strong consensus that fruitful collaboration and high innovation capability result from complementary viewpoints among a diverse group of actors (Mitchell and Nicholas 2006). Rodan and Galunic (2004) imply that heterogeneous knowledge is of high importance to both overall managerial performance and particularly to innovation performance. However, the factual value of diversity and how it should exactly manifest remain unclear. Despite the extensive literature, the role of both similarity and diversity (as opposite ends of a continuum), particularly in the decision-making of choosing academic collaborators, requires more research, as will be shown in what follows. The related work discussed in the previous subsection demonstrates a tendency of utilizing similarity-maximizing approaches for recommending content and connections, thus amplifying the effects of homophily bias. The concept of homophily has caught the attention of researchers, primarily in social psychology (Lazarsfeld and Merton 1954; Marsden 1987; Moody 2001), as the phenomenon of individuals’ natural preference to interact with similar-minded people who share socio-cultural traits. In CSCW and HCI research, homophily has been addressed, for instance, as a predictive and influential factor of online behavior in content preferences (Chang et al. 2014), and audience attraction on social media (Sharma and Cosley 2016). Another vein of research focuses on studying diversity in terms of human, relational and intellectual capital within global organizations to design features that support online communities in collaborative tasks (Muller et al. 2012). Researchers and developers seem to have adopted the similaritymaximizing approach from item recommender systems, using metrics of similarity as the proxy for relevance also in matching peers within organizations (Guy et al. 2010) and scholars in the context of academic collaboration (Heck 2013). Although homophily might strengthen existing communities, it does not encourage the creation of new ties to further away in the global social network. Some researchers propose that such mechanisms directly lead to the formation of echo chambers that are detrimental to information flow, innovation, and creativity (Jasny et al. 2015; Bessi 2016). Echo chambers have received critique particularly with respect to social media services that divide the user community into camps of different opinions and thus increase polarization in the society (Li et al. 2013; Lee et al. 2014). At the same time, organizational studies have identified that also diversity can have negative influences on collaborative activities, such as information exchange and decision-making (Graves and Elsass 2005; Hobman et al. 2004). An extensive review (Mannix and Neale 2005) concludes that social differences (i.e., surfacelevel), such as race and gender, indeed tend to have adverse effects on the ability of groups to function effectively, whereas more profound cognitive dissimilarities, such as differences in expertise or personality, are more often positively related to Scholars’ Perceptions of Relevance... 363 team performance. In other words, diversity is strongly linked to the concept of identity, which can make the introduction of diversity challenging in established work cultures. Following the above mentioned, CSCW research has investigated whether it is possible to overcome the adverse effects of dissimilarities in teams to provoke creativity and productivity. For instance, Dong et al. (2016) found that commitment to a common cause, such as shared goals of the work, bring people together despite cultural differences. Similarly, but from a broader perspective, Ye and Robert Jr. (2017) revealed that collectivism (over individualism) makes people more tolerant to differences in terms of personal values, working styles, skills, and general abilities, thus, embracing individual creativity and work satisfaction. Besides, Rajagopal et al. (2017) investigated how to match peers with dissimilar opinions. The findings demonstrated that matching people with different interpretations of shared interests is more effective in producing positive experiences of breakdown. Overall, the literature on diversity and homophily contains interesting contradictions, which calls for further empirical research on various forms of similarity or diversity in different types of collaboration. To this end, we seek to uncover how the two concepts are interlinked particularly in the assessment of the relevance of potential scholarly collaborators. 2.3. User-centered evaluation criteria for recommender systems Historically, research on recommender system has primarily focused on the design of algorithms, underlying the assumption that better algorithms results in better user experience with the systems. Pu et al. (2012) challenges this premise by providing conceptual observation and guidelines on the evaluation criteria for recommender systems. They explicitly emphasize the importance of the user’s perception regarding the system qualities. We summarize existing conceptualizations of the recommendation quality as follows: (i) perceived accuracy (Pu et al. 2011) – how well recommendations match with users interests defines the trust towards the systems; (ii) familiarity (Sinha and Swearingen 2002)–presenceof familiar items increase trust towards the system; (iii) novelty (Castells et al. 2015) – unexpectedness of received recommendations can affect perceived usefulness of the system; (iv) diversity (Nguyen et al. 2014) – receiving diverse items lessens filter bubble thus increasing users’ satisfaction and, as a consequence, perceived accuracy of the system. Knijnenburg et al. (2012) also provide a framework for the user-centered evaluation of recommender systems that extends the system accuracy metric with other relevant measures. For instance, the authors observe correlations between concepts, such as perceived recommendation quality (relevance), choice satisfaction, variety, diversity, effectiveness, and accuracy along with personal characteristics of the user (e.g., trust towards ICT). 364 Ekaterina Olshannikova et al. To sum up, these types of evaluation criteria focus on subjective user perceptions’ in the evaluation of objective aspects of the system. In this article, we do not question the effectiveness of the designed system and its elements, but rather focus on investigating scholars’ attitudes towards the recommended people as potential collaborators. The evaluation criteria proposed by the prior research has proven to be effective in the assessment of item recommender systems. In contrast, as objects of recommendation, human individuals contain much more diverse features that influence the evaluation. When assisting people in choosing potential collaborators, the subjective perception of relevance might have various facets and be determined by the need or task for partnering, and, therefore, the metrics should be operationalized accordingly. In this article, we approach relevance criteria with the temporal aspect (i.e., from the perspective of past vs. current research interests), as well as from the perspective of potential collaborative activities with the recommendations. 3. System design In this section, we first explicate the choice of the data source used for the design of the recommendation algorithm. Next, we outline the data cleaning process and analysis and, finally, describe the user interface developed for the experiment. 3.1. Data source, data cleaning, and analysis We designed a content-based people recommender system using DBLP, an open bibliographic database of publications records from the majority of Computer Science conferences and journals. The DBLP dataset is a substantial plain ASCII XML file.4The metadata for each record contains more than necessary details for the study and, therefore, requires multiple cleaning procedures. In the first step, the XML file was parsed using the ‘xml.sax’5package targeting on the following tags: article, inproceedings, proceedings, book, incollection, phdthesis, mastersthesis, and www. Then, from the parsed XML file, we extracted 5,847,090 records that consist only of titles, co-authors, publishing years and venues. Next, in the resulted subset we cleaned the titles of publications following three steps: (i) converting letters to lowercase, (ii) removing the English stop words with ‘nltk.corpus.stopwords’6function, (iii) removing the digital strings. The people recommender system runs on the subset of the parsed 5,847,090 records and depends on the input of publication venues of a given user (participant of the study). The detailed data analysis is demonstrated in Figure 1.For a participant, top venues of their publications are given (see step 1 in Figure 1). Then a subset of records is extracted from the parsed DBLP dataset by only those publication venues (step 2). All the titles in the subset records are cleaned as described previously and aggregated to form the corpus profile for each author ever published in those venues (step 3). Those authors who have less than three Scholars’ Perceptions of Relevance... 365 1. Data source and Input Dataset (DBLP) (Author, Title, Year, Conference) Record 1 Record 2 Record n INPUT Participant Name Participant 1 Participant 20 Conference Name Conference 1 Conference n Conference 1 Conference n Low distance recom. (R1, R2, R3) Medium distance recom. (R4, R5, R6) High distance recom. (R7, R8, R9) 8.Output Recommendations 7. Co-Authors Network Filter out co-authors 2. Extract target conference(s) records for Participants Conference 1 (Author (s), Title) Record 1 Record 2 Record m Conference n (Author (s), Title) Record 1 Record 2 Record m 3. Conference 1 (Author, Content) Record 1 Record m Conference n (Author, Content) Record 1 Record m 4. Run TF-IDF TF-IDF Feature Vector Record 1 Record k 5. Sorting Cosine Distances for Participants Distances for Participant 1 Distance 1 Distance i Distances for Participant n Distance 1 Distance i 6. OTSU Filter to identify recommendations of low, medium and high distances 0.0 0 2000 4000 6000 8000 10000 0.2 0.4 0.6 0.8 1.0 End of low distance Mid Point Start of high distance Figure 1. Overview of the DBLP data analysis process to produce people recommendations. publications in recent five years are filtered out to improve the quality of recommendations. Next, we tokenize the corpus and build the vocabulary with the words that only appear once or show up in more than 95% of the author corpus profiles using ‘CountVectorizer()’ function from scikit-learn.7 After corpus tokenization, TF-IDF is applied to the profile model to form feature vectors for each author (step 4). Next, we compute the cosine distances between the given participant and the other authors in the subset records regarding them (step 5). As it is more intuitive to indicate the close distance with a smaller number, we use cosine distance to represent the similarity between two authors. Accordingly, the closest, or the most similar author to a participant will have the smallest cosine distance. To validate the participants’ preferences on similarity-difference continuum during the user study, we decided to deliver recommendations in the form of three groups of controlled distances – low (high similarity), medium (moderate similar) and high (low similarity). To automatically separate recommendations into such groups, the cosine distances between a participant and the other authors are sorted first, and then the OTSU filter (Otsu 1979) is employed to detect the boundaries between each group (step 6). The OTSU filter calculates the optimum threshold separating the two groups so that their intra-class variance is minimal. As the distribution of the cosine distances follows the power law, we implement 372 Ekaterina Olshannikova et al. Distance groupQuestion Low Distance Medium Distance High Distance Low Distance Medium Distance High Distance Low Distance Medium Distance Relevance fo past research interests (Q2) Similarity (Q12) Relevance for current research interests (Q1) High Distance Ranks Low Distance Medium Distance 47.5 35 <.01** <.001*** <.001*** <.001*** <.001*** <.001*** 25.5 <.05* <.01** <.05* 50 34.5 23.5 52 32 24 Test Statistics N=18, Chi-Square=16.5 DF = 2, p<.001*** N=18, Chi-Square=24.03 DF = 2, p<.001*** N=18, Chi-Square=25,21 DF = 2, p<001*** Figure 5. Results of Friedman’s Test and post-hoc analysis. Significance codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05. results, meaning that participants were consistent in their scores independently of the temporal perspective. The quantitative results of perceived similarity (Q12) also demonstrate the tendency of a significant difference in given scores (See Figure 4). Accordingly, matches with low distance are graded as most similar, matches with medium distance as somewhat similar and those with high distance – least similar. This consolidates validity of cosine distances and OTSU filter as a method for identifying thresholds of three degrees of similarity. Friedman test results consolidate a significant difference between recommendations groups (See Figure 5). 5.1.2. Qualitative findings The verbal feedback about relevance and similarity is generally in line with the quantitative results. To provide an overview of participants’ comments, we collected illustrative examples in Table 2, sorted according to the three groups of similarity distance. Although the participants were unaware of the three different similarity distances, in their feedback they distinguish between different degrees of perceived relevance by using phrases like ‘very/most relevant,’ ‘somewhat relevant/not exact match’ and ‘irrelevant/totally irrelevant.’ Feedback about the outliers (in Q1 – 9 cases, in Q2 – 7 cases) – recommendations of high distance rated as relevant – has revealed that such recommendations relate to participants’ current research interests with potential for future directions, or because of surprising topics appeared in their profiles. Such cases hint about interest in dissimilarity and openness for new opportunities, as quotes 1 and 2 illustrate. 1)“(R8) This person has really interesting research topics. [...] I thought it would be fun to get to know something else, something different from my own research focus. I would definitely go to see their poster. Maybe it would be interesting for my future research. This is an interesting and fun recommendation!” (P1, 50 y.o., Finnish female, Post-doc) Scholars’ Perceptions of Relevance... 373 Table 2. Examples of participants’ verbal feedback about given recommendations. Low distance (high similarity) Medium distance (moderate similarity) High distance (low similarity) (R2) I am familiar with this person and his coauthors but not in person because I didn’t collaborate with this Greek team directly. However, I was reviewing their publications, particularly, the most recent paper on the list. I consider this recommendation is the most relevant to me. I would surely like to meet in person. (P3, 32 y.o., Russian male, Senior Researcher) (R4) I guess this person is some research leader or professor. Some papers are close to what I am doing, but not an exact match. I would probably find a lot of interesting discussion with him. He seems to be more experienced than me. There is a clear shared interest. I do not remember hearing about him and do not recognize any of co-authors. (P8, 33 y.o., Finnish male, Postdoctoral Researcher) (R7) I have never seen this person before. There are thousands of young researchers, I am not surprised. I have no idea why the system recommends me this person because his topics are totally different from mine. I do not know any of co-authors and topics do not seem to be relevant. (P7, 56 y.o., Finnish male, Senior Researcher) P3 research interests: wireless technologies, telecommunication and networking P8 research interests: Intenet-of-Things, wireless technology, radio networks, and positioning P7 research interests: Human-computer interaction, VR, gaze technologies and interactions R2 research topics: wireless technology, cellular systems, sensor networks R4 research topics: Neural networks, stochastic decoding R7 research topics: Crowd science, human-computer interaction 2)“(R9) This person is from a different field, and the research has interesting aspects. The last paper in the list is the most interesting: it is about something similar we have been doing recently but not published yet! So, I have to check it and maybe contact the authors.” (P2, 42 y.o, Finnish male, Senior Researcher) Some participants also mentioned that their first impression about recommended people had changed when they started to check the recommendation’s profiles in details (see quote 3). 3) “That was a good idea that you gave me to check all recommendations first because initial reaction was different, but when I started thinking and realized that my first impression maybe was not correct. When you start thinking about recommendations’ relevance further, there might be some changes. So some of them are not that irrelevant as I thought at first.” (P16, 50 y.o., Finnish female, Senior Researcher) Hence, decision-making on perceived relevance was found to be influenced by the participants’ estimation about how topics of recommended people match with their own – whether they are similar, very different or complementary. In 374 Ekaterina Olshannikova et al. this regard, few participants addressed that evaluating the similarity between them and the recommended people is a challenging task. First, they mentioned employing different scales of comparison (5 cases) – assessing a recommended person from the perspective of all research fields in the world or specific focus areas. The second factor refers to the temporal aspect (5 cases): as interests and research directions tend to change, the evaluation of the similarity depends on the chosen time frame. Participants also pointed out that the estimation of relevance and optimal level of similarity for specific collaboration are context dependent (4 cases): some tasks would require cooperation with diverse people, while other tasks benefit from similarity. Thus, when looking for candidates with a distinct set of skills, people should also define shared interests or goals to make a prospective collaboration fruitful. Participants also acknowledged the possible adverse effects of receiving recommendations of people who are very similar (4 cases). For example, in a professional context, high similarity of interests might result in competition, and social interaction with such people will require choosing specific communication strategies. 5.2. Familiarity 5.2.1. Quantitative findings Familiarity variable was evaluated from three perspectives: familiarity with the research topics (Q9); recognizing some of the co-authors’ names (Q10);and knowing the match in person (Q11). The Figure 6depicts that the scores of recommendations with a low distance in all questions are widely distributed, while groups of the medium and high distances received mostly negative ratings. According to scores distribution, participants are mainly familiar with the research of recommendations and aware of many co-authors of recommended people in low distance group. Besides, there are only a few people whom participants know in person. The results of the Friedman test yield a statistically significant difference in given scores (see Figure 7). The post-hoc analysis reveals significant variance only in groups of low vs. medium and low vs. high distances. (Q9) Familiar with the research of a person (Q10) Familiar with co-author(s) (Q11) I know him/her in person 1234567 Participants' score Low Distance Medium Distance High Distance Low Distance Medium Distance High Distance Low Distance Medium Distance High Distance Figure 6. The distribution of participants scores regarding perceived familiarity. Scholars’ Perceptions of Relevance... 375 Distance groupQuestion Low Distance Medium Distance High Distance Low Distance Medium Distance High Distance Low Distance Medium Distance Familiar with co-author(s) (Q10) Know match in person (Q11) Familiar with the research of a person (Q9) High Distance Ranks Low Distance Medium Distance 48 32.5 <.001*** <.01** <.001*** <.001*** <.001*** <.001*** 27.5 .1108 .2597 .7048 46 33.5 28.5 48 30.5 29.5 Test Statistics N=18, Chi-Square=21.26 DF = 2, p<.001*** N=18, Chi-Square=12.04 DF = 2, p<.01** N=18, Chi-Square=23.4 DF = 2, p<.001*** Figure 7. Results of Friedman’s Test and post-hoc analysis. Significance codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05. 5.2.2. Qualitative findings In general, participants were positively surprised to see many unfamiliar people in the list of recommendations and were happy to increase their awareness about new researchers in related scientific fields (see quote 4 and 5). At the same time, they appreciated receiving suggestions about people with established ties or wellknown figures and interpreted it as confirmation of the algorithm validity. 4) I think the 1st one was most familiar, he is a good match. For the others, I started to wonder about their research topics. I tried to see co-authors identifying relation to me and then to topics. I think it is good that the majority of recommendations are unfamiliar because it is a people recommender system. (P6, 35 y.o., Finnish male, Postdoctoral Researcher) 5) I was curious about why I do not know some of the people. I was trying to analyze, and I think there might be a time delay in our academic activities or maybe there is a gap regarding research communities. (P3, 32 y.o., Russian male, Senior Researcher) In the interview, participants addressed that already known people are less exciting recommendations (quotes 6 and 7). Even though we intentionally filtered out all the co-authors, the bibliographic data prevents understanding of the actual social relationships between researchers. Thus, in some cases (9 recommendations out of 54), the system recommended people from very close social circles, for instance, peers from academic projects or colleagues with whom one had not co-authored publications but interact daily (quote 8). 6) “(R3) It seems like I know him. His areas of research match with mine, I would say, by 90%. I know some of his co-authors even in person. This is the most interesting recommendation but not surprising, because I know him and even know his Ph.D. students.” (P5, 42 y.o., Finnish male, Full Professor) 7) “The first and last person in the list I already know. They are obviously relevant recommendations. But the rest is more interesting because I do not know them.” (P10, 40 y.o., Finnish male, Senior Researcher) 376 Ekaterina Olshannikova et al. 8) “(R2) One interesting observation is that one of the recommended people is my roommate. We have similar topics and plenty of other commonalities. We are cooperating mostly by talking. I think, we have never written papers together, but we have shared co-authors.” (P4, 56 y.o., Russian male, Senior Researcher). 5.3. Willingness to interact 5.3.1. Quantitative findings Evaluation of the willingness to interact with recommended people reflects six predefined scenarios of face-to-face interaction or follow-up collaboration at the context of a conference (see Figure 8): (Q3) asking advice, (Q4) giving advice, (Q5) sharing research ideas, (Q6) exploring joint research topics, (Q7) spending time together, and (Q8) organizing a research visit. The distribution of scores demonstrates that participants seemed to have a very positive attitude towards engaging in a low-threshold interaction like sharing research ideas, exploring common research topics and spending time together, particularly with most similar people. Scores given to medium and high distance groups illustrate variance of opinion with a neutral attitude on average. Interestingly, all the six interaction scenarios yielded very similar results, even in the more long-term follow-up action of organizing a research visit. The results of the Friedman test (see Figure 9) depicts a statistically significant difference in given scores only in groups of low vs. high and low vs. medium distance. 5.3.2. Qualitative findings In general, participants have diversified opinions regarding the estimation of any interactions and follow-up activities. The majority of participants (12 cases) (Q3) Ask advice (Q4) Give advice (Q5) Share research ideas 1234567 Participants' score (Q6) Explore joint research topics (Q7) Spend time together (Q8) Organize research visit 1234567 Participants' score Low Distance Medium Distance High Distance Low Distance Medium Distance High Distance Low Distance Medium Distance High Distance Figure 8. The boxplots visualize participants scores distribution about scenarios of willingness to interact. Scholars’ Perceptions of Relevance... 377 Distance groupQuestion Low Distance Medium Distance High Distance Low Distance Medium Distance High Distance Low Distance Low Distance Medium Distance Medium Distance Give advice (Q4) Share research ideas (Q5) Explore joint research topics (Q6) Ask advice (Q3) High Distance High Distance Ranks Low Distance Medium Distance 47 32.5 <.01** <.01** <.01** <.001*** <.001*** <.001*** <.001*** <.001*** 28.5 .3589 1 .2321 .059 45 31.5 31.5 48.5 50.5 32 32.5 27.5 25 Test Statistics N=18, Chi-Square=13.54 DF = 2, p<.001*** N=18, Chi-Square=9.17 DF = 2, p<.01** N=18, Chi-Square=18.453 DF = 2, p<.001*** N=18, Chi-Square=20.818 DF = 2, p<.001*** Low Distance Medium Distance Spend time together (Q7) High Distance <.001*** <.001*** .633 49 30.5 28.5 N=18, Chi-Square=16.754 DF = 2, p<.001*** Low Distance Medium Distance Organize research visit (Q8) High Distance <.001*** <.001*** .0831 50 32 26 N=18, Chi-Square=22.286 DF = 2, p<.001*** Figure 9. Results of Friedman’s Test and post-hoc analysis. Significance codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05. consider such social activities to happen naturally at conferences and do not necessarily require high investments of time in collaboration. For some, it was hard to envision willingness to interact with unfamiliar people based on papers titles and topics of the research. Thus, some participants (8 cases) admitted that they would like to learn more about a recommended person before taking any decisions on social interaction. Some emphasize (5 cases) that even in a real context of visiting the conference, it might be challenging to find relevant people in a crowd and contextualized use of such recommender systems might simplify the process and encourage social interaction with unfamiliar people (see quote 9). 9) “Such system should narrow the focus to serve specific purposes [...] I think about a scenario where I am going to the conference, so then I can define ’show me people who are relevant to this event’ and it would give me a sense of community around it [...] After all these years you sometimes stand somewhere in the corner of a conference hall not knowing anybody. Of course, I can start communicating with random people, but it would be much more efficient if the system can suggest already somewhat relevant people and provide with tickets to talk.” (P11, 53 y.o., Finnish female, Full professor) 5.4. Needs and important factors in research collaboration When specifying crucial needs for collaboration, participants mentioned different activities. The most frequent reasons are understandable when considering 378 Ekaterina Olshannikova et al. senior academics: seeking academic and industrial partners for funding applications (appears in 12 answers) and knowledge sharing as the way of indirect cooperation (12 answers, e.g., exchange of data or finding relevant publications in the topic of interest). Some participants also pointed out a need for people without conflicts of interest (5 answers) such as pre-examiners, reviewers, editors or opponents, who are highly demanded and complicated to find. Another reason was research mobility (5 answers), which calls for cooperation with particularly international universities or companies. Interestingly, whereas similarity was generally considered to be a significant aspect, the above-mentioned collaborative relationships demand heterogeneity of methodological skills, research areas, or social networks. The participants also emphasized that needs for collaboration are occasional and it will be useful to contextualize the recommender system to specific scenarios, for instance, make it particularly locationand event-based or expert-finder. In their opinion, this will ensure the reasoning for using a service and motivate to follow-up on recommendations (see quote 10). 10)“I would be interested in such a system to explore people who are visiting the same conference in advance and filter them based on similarity or relevance. [...] It will help me to revise recommendations faster. Let’s say for the eventbased mobile application it will be great to inform me when a person visits the event and recommend me to meet him there. If a notification to interact comes in the middle of a street, then I doubt it will work. However, if it will happen at the conference I, of course, will try to follow-up on recommendations.” (P8, 33 y.o., Finnish male, Postdoctoral Researcher) The participants also specified factors that matter to them when seeking professional collaboration. First, the majority (14 replies) addressed the importance of affiliation and the current position of candidates. From their perspective, it can tell a lot about the seniority, availability, and potential interest of the people. Besides, considering the relatively high migration of researchers to non-academic positions, it can indicate whether potential cooperators are still pursuing an academic career. Furthermore, many factors can be implicitly obtained from publications. For instance, the quantity, citation rates, and quality of papers might reveal information about the maturity of a researcher, their topics of interests as well as information about their community. For many (8 replies), these aspects play a significant role when aiming to approach unfamiliar scientists (see quote 11). 11) “There are different influence groups with leading experts which are often competitors. So, based on co-authors of a match in his publication lists I can instantly interpret that he belongs to particular influence group. That can help in decision-making whether to collaborate or not and carefully choose the communication strategy.” (P5, 42 y.o., Finnish male, Full Professor) Scholars’ Perceptions of Relevance... 379 Discussion on seniority level brought out various opinions (7 replies). In general, seniority plays a considerable role: for instance, in tasks that require straightforward ability to make decisions (e.g., project planning) it is essential to be in contact with mature researchers, while some practical implementations could be performed in cooperation with students (e.g., assisting a course). Other scenarios might call for open-mindedness regarding this aspect, like in the following example: 12) “The seniority level does not matter to me that much. Sometimes junior people are more creative and innovative. [...] So we should never think about seniority levels. More senior people might have much information, but at the same time too narrow in their vision and interests. Of course, it depends: for consultancy, I might prefer to contact senior people, while for generating new ideas and brainstorming I will be more interested in collaborating with young researchers.” (P15, 50 y.o., Finnish female, Senior Researcher) Additionally, participants address the personal chemistry factor (7 replies), specifically for cases of direct collaboration it might be crucial regarding the efficiency of interpersonal relationships and teamwork (see quote 13): (13) Chemistry plays a significant role – we need some basis for communication. It should be a person with whom it is nice to sit talk and drink coffee in addition to work practicalities. (P15, 65 y.o., British male, Senior Researcher) Thus, participants highlighted that this factor is highly demanded yet unfeasible to be integrated into the system because in their opinion personality compatibility can be assessed only after continuous interactions. 6. Discussion While prior research has aimed at creating meaningful professional connections with the help of people recommender systems, little attention has been put on evaluating the subjective perceptions of the recommendation relevance. We emphasize that recommending is different from predicting new connections (McNee et al. 2006): to design services that can meaningfully enhance professional collaboration, algorithms should go beyond reproducing or strengthening the typical human bias. In the following, we first summarize our findings and reflect on their novelty and relevance. Next, we provide a discussion on limitations and future work. 6.1. Summary of the results We presented the results of an experiment on computer science researchers’ preferences regarding potential collaborators of different similarity levels with a DBLP-based recommender system. With 18 senior scholars in areas related to CS, 380 Ekaterina Olshannikova et al. we tested how the dependent factors of perceived relevance, similarity, familiarity and willingness to interact are related to the independent variable of objective similarity measurement in terms of publication history. The findings reveal that (1) the homophily bias is evident also in scholars’ intuitive assessments of relevance and willingness to interact, and (2) there is a mismatch between people’s intuitive choices and the deliberate intentions in decision-making on potential collaborators. Considering our first research question about which level of system-defined similarity is preferred in participants’ evaluations, the findings demonstrate the highest ratings for most similar people. Methodologically, the subjective evaluations of different similarity levels seem to consolidate the system design, particularly the efficiency of the OTSU filter in identifying different levels on the similarity–difference continuum. In other words, even the relatively simple analytics procedure with scarce data seemed to work sufficiently, and the publication data represented the participants’ topics accurately enough. While the norm in such data analytics tends to stress the need for Big Data (e.g., Hoang et al. 2017), it appears that for people recommendations the systems could suffice with rather simple datasets as long as the recommender engine logic is well designed. The findings imply that the participants were able to retrieve useful suggestions and, for the majority, the evaluation process with the operationalized variables was straightforward. Regarding the second research question about academics’ needs and expectations in professional collaboration, the results demonstrate that the optimal area on the similarity-difference continuum highly depends on the type and context of collaboration. For instance, crucial factors in direct cooperation, such as personal compatibility and similarity of attitudes and beliefs, are not as emphasized in short-term and indirect professional interactions (e.g., consultancy type of cooperation) as in long-term collaboration. Furthermore, the nature of the collaboration task might influence the perceived relevance of potential candidates, for example, regarding the complementarity of professional roles, skills, and knowledge. As a methodological contribution for studying user perceptions, we operationalized the concepts of perceived relevance, similarity, familiarity and willingness to interact as subjective evaluation measures for the context of academic collaboration. This helps to uncover some of the experiential aspects of these concepts and quantitatively assess how individuals consider people recommendations. To complement the reductionist measures, the qualitative findings reveal more complex and nuanced aspects that should be addressed in the design and evaluation of people recommendations for professional partnering. Following the participants’ rationale about important factors in collaboration, we propose that the diversity of recommendations in professional social matching could be enhanced through several dimensions or criteria of relevance: Scholars’ Perceptions of Relevance... 381 –similarity in terms of background, attitudes, values, beliefs, goals and intentions (e.g., research aims). Previous research addressed that similarity of such qualities can raise cohesion or so-called ‘affinity’ (Moreland and Zajonc 1982) in interpersonal relationships, and can even reduce adverse effects of individual dissimilarities (Dong et al. 2016) in collaborative work; –complementarity in terms of professional roles, skills, knowledge, and social capital. In this context, complementarity is beyond pure diversity, as discussed by Mitchell and Nicholas (2006). It should enable relevant opportunities for collaboration by identifying beneficial intersections between individuals’ qualities; –compatibility for direct cooperation in terms of being mentally, socially, morally or emotionally close to each other. This aspect was partially emphasized by Bozeman et al. (2013), who define collaboration as a process of knowledge production, in which compatibility of such qualities can establish trustful, joyful and personally valued cooperation; –approachability/logistics – the availability of a person for direct or indirect interaction in terms of physical proximity as well as social and organizational distance. This dimension echoes with ‘collaboration readiness’ conceptualized by Olson and Olson (2000), who calls for better technological solutions to enable smooth communication and interaction practices for distributed collaboration. 6.2. Limitations and future work We selected a mixed-method approach to enable a broad understanding of our research questions. By combining a controlled experiment with qualitative faceto-face interviews, we intentionally limited the sample size and compromised generalizability with deeper qualitative understanding. In the same vein, the participants represent culturally the same geographical area, which means that the generalizability of the findings to the general population of scholars is limited. Nevertheless, our method allowed us to observe the decision-making process on collaboration in the actual context of using a people recommender system. Additionally, it helped to engage participants in the discussion on potentially relevant partners concerning similarity, complementarity, and other essential qualities or factors when seeking collaboration. We could not have elicited some of the interview findings without having the task of evaluating the recommendations in situ. In fact, our prior research experience suggests that qualitative exploration of human needs, wishes, and expectations often benefits from providing a design artifact that can help a participant to form opinions on abstract concepts and speculated behavior. 388 Ekaterina Olshannikova et al. Moreland, Richard L.; and Robert B. Zajonc (1982). Exposure effects in person perception: Familiarity, similarity, and attraction. vol. 18, no. 5, September 1982, pp. 395–415. Muller, Michael; Kate Ehrlich; Tara Matthews; Adam Perer; Inbal Ronen; and Ido Guy (2012). Diversity among enterprise online communities: collaborating, teaming, and innovating through social media. CHI’12. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Austin, Texas, USA 5–10 May 2012. New York: ACM, pp. 2815–2824. Nguyen, Tien T.; Pik-Mai Hui; F. Maxwell Harper; Loren Terveen; and Joseph A. Konstan (2014). Exploring the filter bubble: the effect of using recommender systems on content diversity. WWW’14. Proceedings of the 23rd international conference on World wide web, Seoul, Korea, 7–11 April 2014. New York: ACM, pp. 677–686. Nishibe, Yoshiyasu; Hiroaki Waki; Ichiro Morihara; Fumio Hattori; Toru Ishida; Toshikazu Nishimura; Hirofumi Yamaki; Takaaki Komura; Nobuyasu Itoh; Tadahiro Gotoh; Toyoaki Nishida; Hideaki Takeda; Atsushi Sawada; Harumi Maeda; Masao Kajihara; and Hidekazu Adachi (1998). Mobile digital assistants for community support. AI Magazine, vol. 19, no. 2, June 1998, pp. 31. Olson, Gary M.; and Judith S. Olson (2000). Distance matters. Human–computer Interaction, vol. 15, no. 2-3, December 2000, pp. 139–178. Otsu, Nobuyuki (1979). A threshold selection method from gray-level histograms. IEEE transactions on systems, man, and cybernetics, vol. 9, no. 1, January 1979, pp. 62–66. Paasovaara, Susanna; Ekaterina Olshannikova; Pradthana Jarusriboonchai; Aris Malapaschas; and Thomas Olsson (2016). Next2You: a proximity-based social application aiming to encourage interaction between nearby people. MUM’16. Proceedings of the 15th International Conference on Mobile and Ubiquitous Multimedia, Rovaniemi, Finland, 12–15 December 2016. New York: ACM, pp. 81–90. Parra, Denis; Wei Jeng; Peter Brusilovsky; Claudia L´ opez; and Shaghayegh Sahebi (2012). Conference Navigator 3: An online social conference support system. UMAP 2012. UMAP workshops at the conference on User Modeling, Adaptation, and Personalization, Montr´eal, Qu´ebec, 16–20 July 2012. Dordrecht, The Netherlands: Kluwer Academic Publishers, pp. 1–4. Pu, Pearl; Li Chen; and Rong Hu (2011). A user-centric evaluation framework for recommender systems. RecSys’11. Proceedings of the fifth ACM conference on Recommender systems, Chicago, Illinois, USA, 23–27 October 2011. New York: ACM, pp. 157–164. Pu, Pearl; Li Chen; and Rong Hu (2012). Evaluating recommender systems from the user’s perspective: survey of the state of the art. User Modeling and User-Adapted Interaction, vol. 22, no. 4-5, October 2012, pp. 317–355. Rajagopal, Kamakshi; Jan M. van Bruggen; and Peter B. Sloep (2017). Recommending peers for learning: Matching on dissimilarity in interpretations to provoke breakdown. British Journal of Educational Technology, vol. 48, no. 2, February 2017, pp. 385–406. Rodan, Simon; and Charles Galunic (2004). More than network structure: How knowledge heterogeneity influences managerial performance and innovativeness. Strategic Management Journal, vol. 25, no. 6, April 2004, pp. 541–562. Salda˜ na, Johnny (2015). The coding manual for qualitative researchers. London: SAGE. Sharma, Amit; and Dan Cosley (2016). Distinguishing between personal preferences and social influence in online activity feeds. CSCW’16. Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work and Social Computing, San Francisco, California, USA, 27 February – 02 March 2016. New York: ACM, pp. 1091–1103. Sheldon, Michael R.; Michael J. Fillyaw; and W. Douglas Thompson (1996). The use and interpretation of the Friedman test in the analysis of ordinal-scale data in repeated measures designs. Physiotherapy Research International, vol. 1, no. 4, November 1996, pp. 221–228. Sie, Rory L. L.; Hendrik Drachsler; Marlies Bitter-Rijpkema; and Peter Sloep (2012). To whom and why should I connect? Co-author recommendation based on powerful and similar peers. Scholars’ Perceptions of Relevance... 389 International Journal of Technology Enhanced Learning (IJTEL), vol. 4, no. 1-2, August 2012, pp. 121–137. Sinha, Rashmi; and Kirsten Swearingen (2002). The role of transparency in recommender systems. CHI EA’02. CHI’02 Extended Abstracts on Human Factors in Computing Systems, Minneapolis, Minnesota, USA, 20 - 25 April 2002. New York: ACM, pp. 830–831. Stanovich, Keith E.; and Richard F. West (2000). Individual differences in reasoning: Implications for the rationality debate? Behavioral and Brain Sciences, vol. 23, no. 5, October 2000, pp. 645–665. Terveen, Loren; and David W. McDonald (2005). Social matching: A framework and research agenda. ACM transactions on computer-human interaction (TOCHI), vol. 12, no. 3, September 2005, pp. 401–434. Tsai, Chun-Hua; and Peter Brusilovsky (2016). A personalized people recommender system using global search approach. IConference 2016. Proceedings of IConference, Philadelphia, Pennsylvania, USA, 20–23 March 2016. iSchools, pp. 1–5. Vassileva, Julita; Gordon McCalla; and Jim Greer (2003). Multi-agent multi-user modeling in IHelp. User Modeling and User-Adapted Interaction, vol. 13, no. 1-2, February 2003, pp. 179– 210. Wongchokprasitti, Chirayu; Peter Brusilovsky; and Denis Parra-Santander (2010). Conference Navigator 2.0: community-based recommendation for academic conferences. SRS ’10. Workshop on Social Reminder Systems, Hong Kong, China 07–07 February 2010. New York: ACM, pp. 1–5. Wuchty, Stefan; Benjamin F. Jones; and Brian Uzzi (2007). The increasing dominance of teams in production of knowledge. Science, vol. 316, no. 5827, May 2007, pp. 1036–1039. Ye, Teng; and Lionel P. Robert Jr. (2017). Does collectivism inhibit individual creativity?: The effects of collectivism and perceived diversity on individual creativity and satisfaction in virtual ideation teams. CSCW’17. Proceedings of the ACM Conference on Computer Supported Cooperative Work and Social Computing, Portland, OR, USA, 25 February – 01 March 2017.New York: ACM, pp. 2344–2358. Yuan, Y. Connie; and Geri Gay (2006). Homophily of network ties and bonding and bridging social capital in computer-mediated distributed teams. Journal of Computer-Mediated Communication, vol. 11, no. 4, October 2006, pp. 1062–1084. Zaiane, Osmar R.; Jiyang Chen; and Randy Goebel (2007). DBconnect: mining research community on DBLP data. WebKDD/SNA-KDD’07. Proceedings of the 9th WebKDD and 1st SNA-KDD 2007 workshop on Web mining and social network analysis, San Jose, California, 12 August 2007. New York: ACM, pp. 74–81. Zenk, Lukas; Michael Smuc; and Florian Windhager (2014). Beyond the name tag. In: B. Lutz (ed.): Wissen nimmt Gestalt an: Beitr¨age zu den Kremser Wissensmanagement-Tagen 2013. Danube, Austria: Edition-Donau-Univ. Krems, pp. 215–224. Zhang, Amy X.; Anant Bhardwaj; and David Karger (2016). Confer: A Conference Recommendation and Meetup Tool. CSCW’16 Companion. Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion, San Francisco, California, USA, 26 February – 2 March 2016. New York: ACM, pp. 118–121. Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.