Mapping Automation in Journalism Studies 2010–2019 : A Literature Review

This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY 4.0 https://creativecommons.org/licenses/by/4.0/ Mapping Automation in Journalism Studies 2010–2019 : A Literature Review © 2023 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group Published version Siitonen, Marko; Laajalahti, Anne; Venäläinen, Päivi Siitonen, M., Laajalahti, A., & Venäläinen, P. (2024). Mapping Automation in Journalism Studies 2010–2019 : A Literature Review. Journalism Studies, 25(3), 299-318. https://doi.org/10.1080/1461670x.2023.2296034 2024 Full Terms & Conditions of access and use can be found at https://www.tandfonline.com/action/journalInformation?journalCode=rjos20 Journalism Studies ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/rjos20 Mapping Automation in Journalism Studies 2010–2019: A Literature Review Marko Siitonen, Anne Laajalahti & Päivi Venäläinen To cite this article: Marko Siitonen, Anne Laajalahti & Päivi Venäläinen (27 Dec 2023): Mapping Automation in Journalism Studies 2010–2019: A Literature Review, Journalism Studies, DOI: 10.1080/1461670X.2023.2296034 To link to this article: https://doi.org/10.1080/1461670X.2023.2296034 © 2023 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group Published online: 27 Dec 2023. Submit your article to this journal Article views: 59 View related articles View Crossmark data Mapping Automation in Journalism Studies 2010–2019: A Literature Review Marko Siitonen a , Anne Laajalahti b and Päivi Venäläinen a a Department of Language and Communication Studies, University of Jyväskylä, Jyväskylä, Finland; b The School of Marketing and Communication, University of Vaasa, Vaasa, Finland ABSTRACT The algorithmic turn has fundamentally transformed journalistic work. Academic interest in the implication of automated algorithms for journalism has grown hand-in-hand with their everyday use. This paper presents a literature review of peerreviewed research reports (N = 62) on automated algorithms in the context of journalistic work. Our review focuses on the first decade (2010–2019) during which automated journalism gained traction. The study identifies the most prominent perspectives or themes that studies in automated journalism have explored and the future directions for research that researchers have proposed. Based on the analysis, the dominant themes that studies in automated journalism have covered include (1) testing and developing algorithmic tools, (2) developing practices and policies for journalistic work, (3) attitudes and technology acceptance, and (4) societal and macro-level discourses concerning AI and journalism. The new directions for research that studies on automated algorithms have recognized relate to (1) target groups and stakeholders—that is, who to study in the future; (2) emergent themes and phenomena—that is, what to study in the future; and (3) approaches and methodologies—that is, how to study these topics in the future. These findings help create a holistic picture of possible future directions for the field. ARTICLE HISTORY Received 31 March 2023 Accepted 12 December 2023 KEYWORDS Algorithmic journalism; artificial intelligence; automated algorithms; automated journalism; computational journalism; robot journalism Introduction During the 2010s, computational or algorithmic journalism, termed “robot journalism” or “automated journalism,” gained increasing traction. On a practical level, companies such as Automated Insights and Narrative Science established early on that automated algorithms can write news articles in fields such as weather, sports, finance, and even education —anywhere where there is a possibility of tapping into well-structured data (Dörr 2016). While the early imaginings of entire newspapers put together by “robot journalists” may not have become commonplace, media organizations worldwide have included aspects of algorithmic journalism into their everyday practices—for example, into collecting © 2023 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/ licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. The terms on which this article has been published allow the posting of the Accepted Manuscript in a repository by the author(s) or with their consent. CONTACT Marko Siitonen [email protected] @MarkoSiitonen JOURNALISM STUDIES https://doi.org/10.1080/1461670X.2023.2296034 and analyzing datasets; identifying trends; producing articles and graphics; and publishing, distributing, and targeting content. Generally, algorithms have changed the manner in which we produce and consume media (Whittaker 2019). Academic interest in the implications of automated algorithms for journalism and journalistic work has grown hand-in-hand with the everyday adoption of such algorithms. The 2010s were a pivotal decade in this regard. For example, it is only toward the latter half of the 2010s that research utilizing naturalistic data from actual working life contexts became available in the field of automated journalism (e.g., Lindén 2017b; Thurman, Dörr, and Kunert 2017). The turn of 2022–2023, on the other hand, has seen a rapid introduction of generative AI such as ChatGPT and Midjourney into the debate. This latest development is something that research is only beginning to catch up with. The starting point of this study is the realization that the continuous developments both in the professional as well as the academic fields related to algorithmic journalism require us to also look back in time in order to construct a holistic overview of where we have been and where we may be heading. Understanding the early stages of research provides valuable historical context and also allows us to trace the evolution of ideas, technologies, and methodologies. This paper presents a literature review of studies that explore the intersection of automated algorithms and journalistic work in the decade between 2010 and 2019. The 2010s are the first decade during which automated algorithms became a realistic option to be included in everyday journalistic work. This study contributes to our understanding of the so-called “algorithmic turn” (Napoli 2014) in the context of journalistic work—what algorithm-based journalistic production can mean for journalism in the years to come. Our interest lies in how scholars have socially constructed the meaning of algorithms in journalistic work—from which viewpoints or perspectives have they studied automated algorithms and what kind of repercussions or opportunities did they see looming ahead. By exploring the boundaries of existing research and what may lie beyond these boundaries, the results of this review will provide directions for future research. The need for review articles has been recently highlighted, as such articles advance theory building and the fields in which they are set (Post et al. 2020). Our study seeks to answer two research questions: RQ1) What were the most prominent perspectives or themes that studies on automated journalism explored in the 2010s? RQ2) What kind of future directions did researchers propose on automated journalism? By answering these questions, our study contributes to the discussion on the past and future of journalism in the digital age. Data and Analysis To achieve our aim, we conducted a literature review on peer-reviewed research reports on automated journalism published between 2010 and 2019. This period was selected after an initial review that revealed that before 2010, literature on the topic was scarce and mainly speculated the potential of automated journalism. Additionally, we decided to limit the review to the end of 2019. Our rationale was that including a full decade of research should allow us to gain in-depth insight into the emerging field, while still keeping the study focused. In conducting the review, we drew on the principles of systematic literature reviews (e.g., Booth, Papaioannou, and Sutton 2012). In summary, we 2 M. SIITONEN ET AL. aimed at an organized and reproducible data collection and analysis process, as well as transparent and explicit reporting of the review and the research findings. We utilized frequently used databases in the fields of humanities, social sciences, and information technology in order to collect a corpus of journal articles that deal with algorithms in the context of journalistic work. The databases that were included in the search were ACM Digital Library, DOAJ, EBSCOhost (Academic Search Elite, Business Source Elite, and Communication and Mass Media Complete), JSTOR, and ProQuest. We also expanded our search on Google Scholar. Since Google Scholar does not offer the same possibilities to limit searches as the other databases, the initial search resulted in thousands of search results. As a solution, Google Scholar searches were restricted to the first 100 results, since the most relevant results appear higher up on the list. Moreover, it must be noted that Google Scholar searches are not as systematic as other databases due to the manner in which Google’s search algorithm personalizes search results. Two separate search strings were used for each database: 1) (“AI” OR “artificial intelligence” OR “robot*” OR “algorithm”) AND “journalis*” 2) “automated journalism” The initial search results included several irrelevant results from the field of medicine, technology, multimedia, and social media studies, despite other search parameters and search terms. As our focus was on the journalists’/professional viewpoint, we concluded that the search term/word “journalis*” would be sufficient to find the most relevant results for our study. After initial scanning, we also included the more specific search term “automated journalism,” since it appeared to have gained sufficient popularity to represent a large proportion of the field and did not necessarily come up using the first search string. In addition, we limited the search to include English language publications only for consistency. The initial search resulted in thousands of possible hits. To narrow down the search results, we scanned the hits produced by the search. During this scanning, we focused on the titles, abstracts, and keywords of each article and, in certain cases, we also read key passages from the main body of the article. Based on this stage, we included studies that approached automated journalism from the viewpoint of journalistic work or analyzed the use of algorithms in actual journalistic practice. In contrast, we were not interested in the audience’s viewpoint, such as personalization algorithms evident in social media from the users’ point of view or how news readers perceive news generated by automated algorithms (e.g., Clerwall 2014; Haim and Graefe 2017; Shin 2021; Wölker and Powell 2018). We also omitted articles detailing news algorithms from the viewpoint of pure information systems development (e.g., technical descriptions of building algorithms and mathematical models). Of the numerous papers detailing prototypes of algorithms, we only included those that tested these prototypes in actual journalistic contexts. We did not include studies that examined the use of automatically generated stories, such as earnings announcements, in other fields, such as business, law, or marketing. Discussions regarding which papers to include and exclude involved the entire research team (see acknowledgements), but the final decision was made by the first author. This helped to keep the selection criteria consistent throughout the process. It must be noted that in many cases the decision was not easy and that there is an inherent JOURNALISM STUDIES 3 element of subjective evaluation in such decision-making. For example, it is not always clear whether original research articles published in conference proceedings or edited volumes have been peer-reviewed or not. After narrowing the focus of the search through the initial scanning, 62 publications were included in the more detailed analysis (Appendix 1). The final sample of publications turned out to be rather diverse. For example, the publication outlets, methods utilized, and keywords selected by the authors varied significantly, as illustrated in Table 1. This initial finding highlights the difficulty of obtaining a holistic overview of the body of studies conducted in relation to automated algorithms in journalistic work. The plethora of partially overlapping terms and keywords may make it difficult for the reader to even locate relevant texts. An examination of the data collection methods represented within the sample revealed that 13 publications included no empirical data. The most common methods included case studies (N = 16) as well as the testing of prototypes or pilot projects (N = 9). Overall, over a third of the sample relied on such data. Approaching stakeholders for their experiences and, in particular, their perceptions was another popular approach. Interviews (N = 11), focus groups (N = 2), surveys (N = 2), and ethnography (N = 3) comprised another one-third or so of the sample. The remainder of the analyzed studies were divided between content analytical methods (N = 4) and analysis of legislation (N = 2). The analysis of the articles was performed in two stages. First, a close reading of the publications helped us construct a holistic view of the data. At this stage, we also created a table to support our analysis process, which served as a means of taking and sharing notes between the authors throughout the process, facilitating our collaboration and helping to ensure the systematicity and quality of the analysis. For each publication, we looked at: (1) the aim of the study; (2) methodology; (3) key theories, models, and concepts; (4) main orientation or perspective of the study; (5) identified directions for future research; and (6) notes on terminology and keywords. After that, due to the other team members pursuing other interests and projects, the first two authors continued the work and engaged in what can be described as a datadriven thematic analysis (King and Brooks 2021; Silverman 2020). Directed by our research questions, we coded recurring patterns in the data. During this second phase of the analysis, emerging themes, and the codes they consisted of were constantly negotiated Table 1. Search process and research data (N = 62). Databases Search strings Inclusion criteria Final sample ACM Digital Library DOAJ EBSCOhost (Academic Search Elite, Business Source Elite, and Communication and Mass Media Complete) JSTOR Google Scholar ProQuest 1. (“AI” OR “artificial intelligence” OR “robot*” OR “algorithm”) AND “journalis*” 2. “automated journalism” Published in the period 2010– 2019 Published in English Article was concentrating on automated journalism from the viewpoint of journalistic work Case study (N = 16) No empirical data (N = 13) Interview (N = 11) Testing of prototypes, pilot projects (N = 9) Content analysis method (N = 4) Ethnography (N = 3) Analysis of legislation (N = 2) Focus group (N = 2) Survey (N = 2) 4 M. SIITONEN ET AL. between the first two authors. Ultimately, our analysis identified four main themes prevalent in studies on automated journalism, and three main thematic areas for future research directions. In the next section, we discuss our findings. First, the most prominent perspectives and themes explored in studies on automated journalism are presented. Thereafter, we discuss the emerging questions proposed by researchers to be addressed in the future. Findings Themes Covered in Previous Studies on Automated Journalism Our first research question addressed what were the most prominent perspectives or themes that studies on automated journalism explored in the 2010s. In the analysis, we focused on issues such as what could be identified as the main aim of the study and what did the authors specifically focus or concentrate on in their argumentation. Based on our analysis, we identified four main themes: 1. Testing and developing algorithmic tools 2. Developing practices and policies for journalistic work 3. Attitudes and technology acceptance 4. Societal and macro-level discourses concerning AI and journalism All the analyzed publications could be categorized as including one or more of the abovementioned perspectives. For example, Carlson’s (2015) study discusses how automated journalism altered journalists’ working practices (theme 2) as well as how it continues to influence the broader understanding of what journalism is or should be (theme 4). In the next few paragraphs, the key findings related to these four themes are explored. Testing and Developing Algorithmic Tools In a field in which there is rapid technological development, it is not surprising to find a large number of studies that utilize testing and developing as their main approach. Studies in this category included a range of approaches from prototype testing (Diakopoulos, De Choudhury, and Naaman 2012) to analyses of existing algorithmic tools (Adair et al. 2017; Leppänen et al. 2017). Moreover, the purposes for which these tools were developed were equally varied. We found studies focused on finding and selecting sources, event-detection, fact-checking, dealing with multilingual data (e.g., machine translation and speech recognition), classification, clustering and assessment of data, niche and geo-specific bots, social media analytics, writing assistants, and so forth. The emphasis on prototyping almost naturally implies that many of these studies were small-scale, short-term, and set in what could be described as laboratory-like conditions. While this is inevitable, it also implies that these early studies are limited in their capacity to inform us of how such tools and applications will fit in and become a part of everyday journalistic workflow after the “new shine” of technology rubs off. JOURNALISM STUDIES 5 Developing Practices and Policies for Journalistic Work The second main theme identified in our analysis was the drive toward developing practices and policies for journalistic work. Studies that included this perspective often offered or discussed manifestos, lists of criteria, general principles, frameworks and so on—both abstract and concrete tools aimed at guiding journalistic work that utilizes automated algorithms. Practices directed toward everyday journalistic work dealt with issues such as selecting, evaluating, or cleaning data. For example, Diakopoulos and Koliska’s (2017) study develops “pragmatic guidelines that facilitate algorithmic transparency” (809). Presented in the form of an empirically grounded typology, they discuss what kind of information could and should be disclosed when using automated algorithms in journalism. Another example is Caswell’s (2016) study which proposes how automated and human contribution to news could be best integrated for the purpose of structuring news. Other studies discussed practices and policies that were clearly aimed at the broader level of media organizations and similar stakeholders. These included policies related to economic considerations and general media ethics (Thurman, Dörr, and Kunert 2017) as well as juridical questions (i.e., copyright, libel, legal liability) (Ombelet, Kuczerawy, and Valcke 2016; Witt 2017). For example, Lewis, Sanders, and Carmody (2019) raise the question of responsibility for the actions of automated algorithms in journalism. Focusing on the US libel law framework, they discuss the difficulties related to determining fault when algorithms are involved as well as how news organizations may (or may not) articulate their defense in case they end up getting sued. In particular, in studies that extend their scope from tangible practices to the policy level, it becomes evident that the ongoing algorithmic turn involves a number of stakeholders beyond news organizations themselves. These include both more obvious actors such as software developers, but also for example legal, educational, and political actors. Attitudes and Technology Acceptance This is the third main area of focus that is apparent in the analyzed studies centered on journalists’ attitudes toward automated algorithms. The interest in attitudes and technology acceptance can clearly be understood as being motivated by the need to understand the sociocultural context of journalistic work. In other words, studies highlighted the need to approach the topic from perspectives other than primarily technological perspectives (e.g., Kim and Kim 2018, 354). Several studies in this category highlighted the need to unpack the so-called technology acceptance challenges and “automation anxiety” (Lindén 2017a). As Lindén (2017b) notes, journalists’ stance toward new technology has always had its frictions. Whether labeled computer-anxiety or a general fear of technology, it is not difficult to find those who consider automation as a threat to the profession. In certain cases, authors adopted evaluative positions—for example, stating that algorithms could never replace humans as guardians of democracy and human rights (Latar 2015). Even in cases in which it cannot be termed actual “fear,” several studies highlighted journalists’ doubts and disillusionment with the new technology: “Journalists felt these constraints meant that items produced in this way would lack the context, complexity, and creativity of traditional reporting” (Thurman, Dörr, and Kunert 2017, 1246). Another example of such 6 M. SIITONEN ET AL. apprehensions is found in van der Kaa and Krahmer’s (2014) study, in which they note that, “In our experiment, journalists perceived the trustworthiness of a journalist to be much higher than that of the computer” (1). However, not all the viewpoints presented in studies on attitudes and technology acceptance were negative. In certain cases, studies illustrated how journalistic pieces authored by algorithms could be rated higher than human-written ones by both lay readers as well as journalists (Jung et al. 2017). Other studies highlighted that in addition to negative perspectives, there are also those within the journalistic profession who have more positive expectations from this automation (Kim and Kim 2018). Overall, studies in this category build a strong case for continuing to study professionals’ attitudes and the way they incorporate automated algorithms into their work. Societal and Macro-Level Discourses Concerning AI and Journalism The fourth theme our analysis identified was centered on the societal and macro-level discourses surrounding automated algorithms and journalistic work. Within this category of studies, scholars imagined the future of automated algorithms in journalistic work by discussing the impact of algorithmic authorship (Montal and Reich 2017), algorithmic transparency (Diakopoulos and Koliska 2017), legal repercussions (Witt 2017), and how the quantitative turn requires the stakeholders in journalism to acknowledge and answer new ethical questions (Dörr and Hollnbuchner 2017). Unlike in the previous categories where the focus was often on journalistic work, studies included in the fourth theme sought to elevate the discussion to much broader questions. For example, Latar (2015) asserted that “robot journalists” could never replace humans as the “guardians of democracy and human rights” (4). In their most philosophical form, studies in this category attempted to address ontological (Primo and Zago 2015) and epistemic (Parasie 2015; Steensen 2019) reorientations of journalism. There is clearly a set of deeper questions here, identified by scholars such as Stray (2019), who proposed that “One key inter-disciplinary question is the algorithmic description of what counts as news” (2). This challenge was identified by others as well. Carlson (2019) highlighted how automated algorithms in journalistic work would not only fit existing models of news, but also change how news can be imagined. van Dalen (2012) noted how the journalistic profession has often had to come up with redefinitions of what journalism is and that journalists would surely attempt to maintain their position as being in control of “news.” Overall, many of the more philosophical takes on the future of automated algorithms in journalistic work carried a streak of foreboding: If the institutions and professionals of journalism do not update their information literacy competencies, and if the public doesn’t have faith in journalism’s ability to master such competencies, journalism will lose its societal relevance, simply because it loses its ability to produce trustworthy knowledge. (Steensen 2019, 185) What makes such questions particularly difficult to tackle is the realization that neither journalists nor any other actor can answer these questions in isolation. While exploring the four main themes, a few “weak signals” were detected as well. The first one is concerned with the way the field has developed. According to our analysis, during the 2010s, the discussion on automated algorithms in journalistic work shifted JOURNALISM STUDIES 7 References Adair, B., C. Li, J. Yang, and C. Yu. 2017. “Progress Toward ‘the Holy Grail’: The Continued Quest to Automate Fact-Checking.” Computation + Journalism Symposium, October 2017, Evanston, Illinois, United States. Bijker, W. E., T. P. Hughes, and T. Pinch, eds. 1987. The Social Construction of Technological Systems: New Directions in the Sociology and History of Technology, vol. 17. Cambridge, MA: MIT Press. Booth, A., D. Papaioannou, and A. Sutton. 2012. Systematic Approaches to a Successful Literature Review. London: Sage. Broussard, M., N. Diakopoulos, A. L. Guzman, R. Abebe, M. Dupagne, and C.-H. Chuan. 2019. “Artificial Intelligence and Journalism.” Journalism & Mass Communication Quarterly 96 (3): 673–695. https:// doi.org/10.1177/1077699019859901. Carlson, M. 2015. “The Robotic Reporter: Automated Journalism and the Redefinition of Labor, Compositional Forms, and Journalistic Authority.” Digital Journalism 3 (3): 416–431. https://doi. org/10.1080/21670811.2014.976412. Carlson, M. 2017. “Automated Journalism: A Posthuman Future for Digital News?” In The Routledge Companion to Digital Journalism Studies, edited by B. Franklin and S. Eldridge II, 226–234. Taylor and Francis. https://doi.org/10.4324/9781315713793 Carlson, M. 2019. “News Algorithms, Photojournalism and the Assumption of Mechanical Objectivity in Journalism.” Digital Journalism 7 (8): 1117–1133. https://doi.org/10.1080/21670811.2019. 1601577. Caswell, D. 2016. “Computable News Ecosystems: Roles for Humans and Machines.” Proceedings of 2nd Workshop on Computing News Storylines, November 2016, Austin, Texas, United States. Chyi, H. I., S. C. Lewis, and N. Zheng. 2016. “Parasite or Partner? Coverage of Google News in an Era of News Aggregation.” Journalism & Mass Communication Quarterly 93 (4): 789–815. https://doi.org/ 10.1177/1077699016629370. Clerwall, C. 2014. “Enter the Robot Journalist: Users’ Perceptions of Automated Content.” Journalism Practice 8 (5): 519–531. https://doi.org/10.1080/17512786.2014.883116. de Araujo, L. V. 2018. “Algorithms, Artificial Intelligence and NLG in the Production of Brazilian Journalism.” Set International Journal of Broadcast Engineering. https://doi.org/10.18580/setijbe. 2018.11. DeVerna, M. R., H. Y. Yan, K.-C. Yang, and F. Menczer. 2023. “Artificial Intelligence is Ineffective and Potentially Harmful for Fact Checking.” arXiv. https://doi.org/10.48550/arXiv.2308.10800. Diakopoulos, N. 2017. “Enabling Accountability of Algorithmic Media: Transparency as a Constructive and Critical Lens.” In Transparent Data Mining for big and Small Data, edited by T. Cerquitelli, D. Quercia, and F. Pasquale, 25–43. Studies in Big Data 32. Springer. https://doi.org/ 10.1007/978-3-319-54024-5_2 Diakopoulos, N., M. De Choudhury, and M. Naaman. 2012. “Finding and Assessing Social Media Information Sources in the Context of Journalism.” CHI: Conference on Human Factors in Computing Systems, May 2012, Austin, Texas, USA. Diakopoulos, N., and M. Koliska. 2017. “Algorithmic Transparency in the News Media.” Digital Journalism 5 (7): 809–828. https://doi.org/10.1080/21670811.2016.1208053. Dörr, K. N. 2016. “Mapping the Field of Algorithmic Journalism.” Digital Journalism 4 (6): 700–722. https://doi.org/10.1080/21670811.2015.1096748. Dörr, K. N., and K. Hollnbuchner. 2017. “Ethical Challenges of Algorithmic Journalism.” Digital Journalism 5 (4): 404–419. https://doi.org/10.1080/21670811.2016.1167612. Haim, M., and A. Graefe. 2017. “Automated News: Better Than Expected?” Digital Journalism 8 (5): 1044–1059. https://doi.org/10.1080/21670811.2017.1345643. Jesson, J. K., L. Matheson, and F. M. Lacey. 2011. Doing Your Literature Review: Traditional and Systematic Techniques. London: Sage. Jung, J., H. Song, Y. Kim, H. Im, and S. Oh. 2017. “Intrusion of Software Robots into Journalism: The Public’s and Journalists’ Perceptions of News Written by Algorithms and Human Journalists.” Computers in Human Behavior 71: 291–298. https://doi.org/10.1016/j.chb.2017.02.022. 14 M. SIITONEN ET AL. Kim, D., and S. Kim. 2018. “Newspaper Journalists’ Attitudes Towards Robot Journalism.” Telematics and Informatics 35 (2): 340–357. https://doi.org/10.1016/j.tele.2017.12.009. King, N., and J. Brooks. 2021. “Thematic Analysis in Organizational Research.” In The SAGE Handbook of Qualitative Business and Management Research Methods, edited by C. Cassell, A. L. Cunliffe, and G. Grandy, 219–236. London: Sage. https://doi.org/10.4135/9781526430236 Latar, N. L. 2015. “The Robot Journalist in the Age of Social Physics: The End of Human Journalism?” In The New World of Transitioned Media: Digital Realignment and Industry Transformation, edited by G. Einav, 65–80. Cham: Springer. Latour, B. 2005. Reassembling the Social: An Introduction to Actor-Network-Theory. Oxford: Oxford University Press. Leppänen, L., M. Munezero, M. Granroth-Wilding, and H. Toivonen. 2017. “Data-Driven News Generation for Automated Journalism.” Proceedings of the 10th International Natural Language Generation conference (pp. 188–197), September 2017, Association for Computational Linguistics, Santiago de Compostela, Spain. Lewis, S. C., A. K. Sanders, and C. Carmody. 2019. “Libel by Algorithm? Automated Journalism and the Threat of Legal Liability.” Journalism & Mass Communication Quarterly 96 (1): 60–81. https://doi.org/10.1177/1077699018755983. Lindén, C.-G. 2017a. “Algorithms for Journalism: The Future of News Work.” The Journal of Media Innovations 4 (1): 60–76. https://doi.org/10.5617/jmi.v4i1.2420. Lindén, C.-G. 2017b. “Decades of Automation in the Newsroom.” Digital Journalism 5 (2): 123–140. https://doi.org/10.1080/21670811.2016.1160791. Lindén, C.-G., and H. Tuulonen, eds. 2019. News Automation: The Rewards, Risks and Realities of ‘Machine Journalism’. WAN-IFRA Report. Frankfurt: World Association of Newspapers and News Publishers. Loosen, W., M. Häring, Z. Kurtanović, L. Merten, J. Reimer, L. van Roessel, and W. Maalej. 2017. “Making Sense of User Comments: Identifying Journalists’ Requirements for a Comment Analysis Framework.” Studies in Communication and Media 6: 333–364. https://doi.org/10.5771/ 2192-4007-2017-4-333. Mackay, H. 2017. “Social Media Analytics: Implications for Journalism and Democracy.” Journal of Information Ethics 26 (1): 34–48. Maiden, N., K. Zachos, A. Brown, G. Brock, L. Nyre, A. Nygård Tonheim, D. Apostolou, and J. Evans. 2018. “Making the News: Digital Creativity Support for Journalists.” CHI 2018, April 21–26, 2018, Montreal, QC, Canada Association for Computing Machinery. https://doi.org/10.1145/ 3173574.3174049. Milosavljević, M., and I. Vobič. 2019. “Human Still in the Loop: Editors Reconsider the Ideals of Professional Journalism Through Automation.” Digital Journalism 7 (8): 1098–1116. https://doi. org/10.1080/21670811.2019.1601576. Miroshnichenko, A. 2018. “AI to Bypass Creativity. Will Robots Replace Journalists? (The Answer is “yes”.”. Information 9 (7): 183. https://doi.org/10.3390/info9070183. Missaoui, S., M. Gutierrez-Lopez, A. MacFarlane, S. Makri, C. Porlezza, and G. Cooper. 2019. “How to Blend Journalistic Expertise with Artificial Intelligence for Research and Verifying News Stories.” CHI 2019 ACM Conference on Human Factors in Computing Systems, 04–09 May 2019, Glasgow, UK. Montal, T., and Z. Reich. 2017. “I, Robot. You, Journalist. Who is the Author?” Digital Journalism 5 (7): 829–849. https://doi.org/10.1080/21670811.2016.1209083. Napoli, P. M. 2014. “Automated Media: An Institutional Theory Perspective on Algorithmic Media Production and Consumption.” Communication Theory 24 (3): 340–360. https://doi.org/10.1111/ comt.12039. O’Brien, A. M., and C. Mc Guckin. 2016. The Systematic Literature Review Method: Trials and Tribulations of Electronic Database Searching at Doctoral Level. SAGE Research Methods Cases. London: Sage. Ombelet, P. J., A. Kuczerawy, and P. Valcke. 2016. “Employing Robot Journalists: Legal Implications, Considerations and Recommendations.” WWW ‘16 Companion: Proceedings of the 25th International Conference Companion on World Wide Web, pp. 731–736, April 2016, Montréal, Québec, Canada. https://doi.org/10.1145/2872518.2890093. JOURNALISM STUDIES 15 Parasie, S. 2015. “Data-driven Revelation?” Digital Journalism 3 (3): 364–380. https://doi.org/10.1080/ 21670811.2014.976408. Pentzold, C., and D. Fechner. 2021. “Probabilistic Storytelling and Temporal Exigencies in Predictive Data Journalism.” Digital Journalism 9 (6): 715–736. https://doi.org/10.1080/21670811.2021. 1878920. Perreault, G. P. 2020. “What is Digital Journalism? Defining the Practice and Role of the Digital Journalist.” Digital Journalism 8 (10): 1298–1316. https://doi.org/10.1080/21670811.2020. 1851279. Post, C., R. Sarala, C. Gatrell, and J. E. Prescott. 2020. “Advancing Theory with Review Articles.” Journal of Management Studies 57 (2): 351–376. https://doi.org/10.1111/joms.12549. Postăvaru, G., and D. Cramer. 2016. A Case of Methodological Premises Underlying Literature Reviews. SAGE Research Methods Cases. London: Sage. Primo, A., and G. Zago. 2015. “Who and What do Journalism?” Digital Journalism 3 (1): 38–52. https:// doi.org/10.1080/21670811.2014.927987. Ratcliffe, S. 2016. Oxford Essential Quotations. 4th Ed. Oxford University Press. https://www. oxfordreference.com/view/10.1093acref/9780191826719.001.0001/q-oro-ed4-00018679 Ryfe, D. 2022. “Actor-Network Theory and Digital Journalism.” Digital Journalism 10 (2): 267–283. https://doi.org/10.1080/21670811.2021.1945937. Savela, N., T. Turja, and A. Oksanen. 2018. “Social Acceptance of Robots in Different Occupational Fields: A Systematic Literature Review.” International Journal of Social Robotics 10: 493–502. https://doi.org/10.1007/s12369-017-0452-5. Shin, D. 2021. “Why Does Explainability Matter in News Analytic Systems? Proposing Explainable Analytic Journalism.” Journalism Studies 22 (8): 1047–1065. https://doi.org/10.1080/1461670X. 2021.1916984. Silverman, D. 2020. Interpreting Qualitative Data: Methods for Analyzing Talk, Text and Interaction. 6th ed. Thousand Oaks, CA: Sage. Steensen, S. 2019. “Journalism’s Epistemic Crisis and its Solution: Disinformation, Datafication and Source Criticism.” Journalism 20 (1): 185–189. https://doi.org/10.1177/1464884918809271. Stray, J. 2019. “Making Artificial Intelligence Work for Investigative Journalism.” Digital Journalism 7 (8): 1076–1097. https://doi.org/10.1080/21670811.2019.1630289. Thurman, N., K. Dörr, and J. Kunert. 2017. “When Reporters Get Hands-on with Robo-Writing: Professionals Consider Automated Journalism’s Capabilities and Consequences.” Digital Journalism 5 (10): 1240–1259. https://doi.org/10.1080/21670811.2017.1289819. van Dalen, A. 2012. “The Algorithms Behind the Headlines.” Journalism Practice 6 (5–6): 648–658. https://doi.org/10.1080/17512786.2012.667268. van der Kaa, H. A. J., and E. J. Krahmer. 2014. “Journalist versus News Consumer: The Perceived Credibility of Machine Written News.” Proceedings of the Computation + Journalism conference, October 2014, Columbia University, New York, United States. Wang, S. 2021. “Moderating Uncivil User Comments by Humans or Machines? The Effects of Moderation Agent on Perceptions of Bias and Credibility in News Content.” Digital Journalism 9 (1): 64–83. https://doi.org/10.1080/21670811.2020.1851279. Whittaker, J. P. 2019. Tech Giants, Artificial Intelligence, and the Future of Journalism. New York: Routledge. Witt, L. 2017. “Preventing the Rogue bot Journalist: Protection from Non-Human Defamation.” Colorado Technology Law Journal 15 (2): 517–548. Wölker, A., and T. E. Powell. 2018. “Algorithms in the Newsroom? News Readers’ Perceived Credibility and Selection of Automated Journalism.” Journalism 22 (1): 1–18. https://doi.org/10. 1177/146488491875707. 16 M. SIITONEN ET AL. Appendix 1: Research Reports Included in the Review (N = 62) Adair, B., C. Li, J. Yang, and C. Yu. 2017. “Progress Toward ‘the Holy Grail’: The Continued Quest to Automate Fact-Checking.” Computation + Journalism Symposium, October 2017, Evanston, Illinois, United States. Adair, B., M. Stencel, C. Clabby, and C. Li. 2019. “Fact-Checking Automation with a Human Touch: How People Can Help Algorithms Expand the Production of Accountability Journalism.” Computation + Journalism Symposium, February 2019, Miami, Florida, United States. Alnajjar, K., L. Leppänen, and H. Toivonen. 2019. “No Time like the Present: Methods for Generating Colourful and Factual Multilingual News Headlines.” Proceedings of the 10th International Conference on Computational Creativity 2019, June 17–21, 2019, North Carolina, USA. Blankespoor, E., E. deHaan, and C. Zhu. 2018. “Capital Market Effects of Media Synthesis and Dissemination: Evidence from Robo-Journalism.” Review of Accounting Studies 23: 1–36. https://doi. org/10.1007/s11142-017-9422-2. Boyles, J. L., and J. Meisinger. 2020. “Automation and Adaptation: Reshaping Journalistic Labor in the Newsroom Library.” Convergence: The International Journal of Research Into New Media Technologies 26 (1): 178–192. https://doi.org/10.1177/1354856518772542. Brlek, S. S., J. Smrke, and I. Vobič. 2017. “Engineering Technologies for Journalism in the Digital age.” Digital Journalism 5 (8): 1025–1043. https://doi.org/10.1080/21670811.2017.1338526. Broussard, M. 2015. “Artificial Intelligence for Investigative Reporting.” Digital Journalism 3 (6): 814– 831. https://doi.org/10.1080/21670811.2014.985497. Cao, T.-D., I. Manolescu, and X. Tannier. 2018. “Searching for Truth in a Database of Statistics.” WebDB’18: 21st International Workshop on the Web and Databases, June 2018, Houston, Texas, United States. https://doi.org/10.1145/3201463.3201467. Carlson, M. 2015. “The Robotic Reporter: Automated Journalism and the Redefinition of Labor, Compositional Forms, and Journalistic Authority.” Digital Journalism 3 (3): 416–431. https://doi.org/ 10.1080/21670811.2014.976412. Carlson, M. 2017. “Automated Journalism: A Posthuman Future for Digital News?” In The Routledge Companion to Digital Journalism Studies, edited by B. Franklin and S. Eldridge II, 226–234. Taylor and Francis. https://doi.org/10.4324/9781315713793. Carlson, M. 2018. “Automating Judgment? Algorithmic Judgment, News Knowledge, and Journalistic Professionalism.” New Media & Society 20 (5): 1755–1772. https://doi.org/10.1177/ 1461444817706684. Carlson, M. 2019. “News Algorithms, Photojournalism and the Assumption of Mechanical Objectivity in Journalism.” Digital Journalism 7 (8): 1117–1133. https://doi.org/10.1080/21670811.2019. 1601577. Caswell, D. 2016. “Computable News Ecosystems: Roles for Humans and Machines.” Proceedings of 2nd Workshop on Computing News Storylines, November 2016, Austin, Texas, United States. Caswell, D., and K. Dörr. 2019. “Automating Complex News Stories by Capturing News Events as Data.” Journalism Practice 13 (8): 951–955. https://doi.org/10.1080/17512786.2019.1643251. Chyi, H. I., S. C. Lewis, and N. Zheng. 2016. “Parasite or Partner? Coverage of Google News in an era of News Aggregation.” Journalism & Mass Communication Quarterly 93 (4): 789–815. https://doi. org/10.1177/1077699016629370. Cohen, N. S. 2015. “From Pink Slips to Pink Slime: Transforming Media Labor in a Digital Age.” The Communication Review 18: 98–122. https://doi.org/10.1080/10714421.2015.1031996. de Araujo, L. V. 2018. “Algorithms, Artificial Intelligence and NLG in the Production of Brazilian Journalism.” Set International Journal of Broadcast Engineering. http://doi.org/10.18580/ setijbe.2018.11. Diakopoulos, N. 2015. “Algorithmic Accountability.” Digital Journalism 3 (3): 398–415. https://doi. org/10.1080/21670811.2014.976411. Diakopoulos, N. 2017. “Enabling Accountability of Algorithmic Media: Transparency as a Constructive and Critical Lens.” In Transparent Data Mining for big and Small Data, edited by JOURNALISM STUDIES 17 T. Cerquitelli, D. Quercia, and F. Pasquale, 25–43. Studies in Big Data 32. Springer. https://doi. org/10.1007/978-3-319-54024-5_2. Diakopoulos, N., M. De Choudhury, and M. Naaman. 2012. “Finding and Assessing Social Media Information Sources in the Context of Journalism.” CHI: Conference on Human Factors in Computing Systems, May 2012, Austin, Texas, USA. Diakopoulos, N., and M. Koliska. 2017. “Algorithmic Transparency in the News Media.” Digital Journalism 5 (7): 809–828. https://doi.org/10.1080/21670811.2016.1208053. Dierickx, L. 2019. “Automated News Production within the Uses and Professional Practices.” VI Congresso Internacional de Ciberjornalismo, 234–249. November 2018, Porto, Portugal. Dörr, N. K. 2016. “Mapping the Field of Algorithmic Journalism.” Digital Journalism 4 (6): 700–722. https://doi.org/10.1080/21670811.2015.1096748. Dörr, K. N., and K. Hollnbuchner. 2017. “Ethical Challenges of Algorithmic Journalism.” Digital Journalism 5 (4): 404–419. https://doi.org/10.1080/21670811.2016.1167612. Fanta, A. 2017. “Putting Europe’s Robots on the Map: Automated Journalism in News Agencies.” Reuters Institute Fellowship Paper. University of Oxford. Galily, Y. 2018. “Artificial Intelligence and Sports Journalism: Is it a Sweeping Change?” Technology in Society 54: 47–51. https://doi.org/10.1016/j.techsoc.2018.03.001. Johri, A., E.-H. Han, and D. Mehta. 2016. “Domain Specific Newsbots: Live Automated Reporting Systems Involving Natural Language Communication.” Computational Journalism Conference. September 2016, Stanford, CA, USA. Jung, J., H. Song, Y. Kim, H. Im, and S. Oh. 2017. “Intrusion of Software Robots into Journalism: The Public’s and Journalists’ Perceptions of News Written by Algorithms and Human Journalists.” Computers in Human Behavior 71: 291–298. https://doi.org/10.1016/j.chb.2017.02.022. Kim, D., and S. Kim. 2018. “Newspaper Journalists’ Attitudes Towards Robot Journalism.” Telematics and Informatics 35 (2): 340–357. https://doi.org/10.1016/j.tele.2017.12.009. Koliska, M., and N. Diakopoulos. 2018. “Disclose, Decode, and Demystify: An Empirical Guide to Algorithmic Transparency.” In The Routledge Handbook of Developments in Digital Journalism Studies, edited by S. A. Eldridge and B. Franklin, 251–264. New York: Taylor and Francis. https://doi.org/ 10.4324/9781315270449-20. Latar, N. L. 2015. “The Robot Journalist in the Age of Social Physics: The End of Human Journalism?” In The New World of Transitioned Media: Digital Realignment and Industry Transformation, edited by G. Einav, 65–80. Cham: Springer. Leppänen, L., M. Munezero, M. Granroth-Wilding, and H. Toivonen. 2017. “Data-Driven News Generation for Automated Journalism.” Proceedings of the 10th International Natural Language Generation conference (188–197), September 2017, Association for Computational Linguistics, Santiago de Compostela, Spain. Lewis, S. C., A. K. Sanders, and C. Carmody. 2019. “Libel by Algorithm? Automated Journalism and the Threat of Legal Liability.” Journalism & Mass Communication Quarterly 96 (1): 60–81. https://doi.org/10.1177/1077699018755983. Lindén, C.-G. 2017a. “Algorithms for Journalism: The Future of News Work.” The Journal of Media Innovations 4 (1): 60–76. http://dx.doi.org/10.5617/jmi.v4i1.2420. Lindén, C.-G. 2017b. “Decades of Automation in the Newsroom.” Digital Journalism 5 (2): 123–140. https://doi.org/10.1080/21670811.2016.1160791. Lokot, T., and N. Diakopoulos. 2016. “News Bots.” Digital Journalism 4 (6): 682–699. https://doi.org/ 10.1080/21670811.2015.10818. Loosen, W., M. Häring, Z. Kurtanović, L. Merten, J. Reimer, L. van Roessel, and W. Maalej. 2017. “Making Sense of User Comments: Identifying Journalists’ Requirements for a Comment Analysis Framework.” Studies in Communication and Media 6: 333–364. https://doi.org/10.5771/21924007-2017-4-333. Mackay, H. 2017. “Social Media Analytics: Implications for Journalism and Democracy.” Journal of Information Ethics 26 (1): 34–48. Maiden, N., K. Zachos, A. Brown, G. Brock, L. Nyre, A. Nygård Tonheim, D. Apostolou, and J. Evans. 2018. “Making the News: Digital Creativity Support for Journalists.” CHI 2018, April 21–26, 18 M. SIITONEN ET AL. 2018, Montreal, QC, Canada Association for Computing Machinery. https://doi.org/10.1145/ 3173574.3174049. Milosavljević, M., and I. Vobič. 2019a. “Human Still in the Loop: Editors Reconsider the Ideals of Professional Journalism Through Automation.” Digital Journalism 7 (8): 1098–1116. https://doi.org/ 10.1080/21670811.2019.1601576. Milosavljević, M., and I. Vobič. 2019b. ““Our Task is to Demystify Fears’: Analyzing Newsroom Management of Automation in Journalism.” Journalism 22 (9): 2203–2221. https://doi.org/10.1177/ 1464884919861598. Miranda, S., D. Nogueira, A. Mendes, A. Vlachos, A. Secker, R. Garrett, J. Mitchel, and Z. Marinho. 2019. “Automated Fact Checking in the News Room.” WWW ‘19 (3579–3583), May 13–17, 2019, San Francisco, CA, USA. Miroshnichenko, A. 2018. “AI to Bypass Creativity. Will Robots Replace Journalists? (The Answer is ‘yes’.” Information 9 (7): 183. https://doi.org/10.3390/info9070183. Missaoui, S., M. Gutierrez-Lopez, A. MacFarlane, S. Makri, C. Porlezza, and G. Cooper. 2019. ‘How to Blend Journalistic Expertise with Artificial Intelligence for Research and Verifying News Stories.” CHI 2019 ACM Conference on Human Factors in Computing Systems, May 4–9, 2019, Glasgow, UK. Montal, T., and Z. Reich. 2017. “I, Robot. You, Journalist. Who is the Author?” Digital Journalism 5 (7): 829–849. https://doi.org/10.1080/21670811.2016.1209083. Monti, M. 2018. “Automated Journalism and Freedom of Information: Ethical and Juridical Problems Related to AI in the Press Field.” Opinio Juris in Comparatione 1 (1): 139–155. https://ssrn.com/ abstract=3318460. Ombelet, P. J., A. Kuczerawy, and P. Valcke. 2016. “Employing Robot Journalists: Legal Implications, Considerations and Recommendations.” WWW ‘16 Companion: Proceedings of the 25th International Conference Companion on World Wide Web, pp. 731–736, April 2016, Montréal, Québec, Canada. https://doi.org/10.1145/2872518.2890093. Parasie, S. 2015. “Data-Driven Revelation?” Digital Journalism 3 (3): 364–380. https://doi.org/10.1080/ 21670811.2014.976408. Primo, A., and G. Zago. 2015. “Who and What do Journalism?” Digital Journalism 3 (1): 38–52. https:// doi.org/10.1080/21670811.2014.927987. Sirén-Heikel, S., L. Leppänen, C.-G. Lindén, and A. Bäck. 2019. “Unboxing News Automation: Exploring Imagined Affordances of Automation in News Journalism.” Nordic Journal of Media Studies 1: 47–66. https://doi.org/10.2478/njms-2019-0004. Spielkamp, M. 2017. “AlgorithmWatch: What Role Can a Watchdog Organization Play in Ensuring Algorithmic Accountability?” In Transparent Data Mining for big and Small Data. Studies in big Data, edited by T. Cerquitelli, D. Quercia, and F. Pasquale, vol. 32, 207–215. Cham: Springer. https://doi.org/10.1007/978-3-319-54024-5_9. Steensen, S. 2019. “Journalism’s Epistemic Crisis and its Solution: Disinformation, Datafication and Source Criticism.” Journalism 20 (1): 185–189. https://doi.org/10.1177/1464884918809271. Strauss, N. 2019. “Financial Journalism in Today’s High-Frequency News and Information Era.” Journalism 20 (2): 274–291. https://doi.org/10.1177/1464884917753556. Stray, J. 2019. “Making Artificial Intelligence Work for Investigative Journalism.” Digital Journalism 7 (8): 1076–1097. https://doi.org/10.1080/21670811.2019.1630289. Thurman, N., K. Dörr, and J. Kunert. 2017. “When Reporters Get Hands-on with Robo-Writing.” Digital Journalism 5 (10): 1240–1259. https://doi.org/10.1080/21670811.2017.1289819. Rojas Torrijos, F. L. 2019. “Automated Sports Coverage. Case Study of bot Released by The Washington Post During the Río 2016 and PyeongChang 2018 Olympics.” Revista Latina de Comunicación Social 74: 1729–1747. https://doi.org/10.4185/RLCS-2019-1407-90en. van Dalen, A. 2012. “The Algorithms Behind the Headlines.” Journalism Practice 6 (5–6): 648–658. https://doi.org/10.1080/17512786.2012.667268. van der Kaa, H. A. J., and E. J. Krahmer. 2014. “Journalist versus News Consumer: The Perceived Credibility of Machine Written News.” Proceedings of the Computation + Journalism conference, October 2014, Columbia University, New York, United States. JOURNALISM STUDIES 19 Veglis, A., and T. A. Maniou. 2019. “Chatbots on the Rise: A new Narrative in Journalism.” Studies in Media and Communication 7 (1): 1–6. https://doi.org/10.11114/smc.v7i1.3986. Vrochidis, S., A. Moumtzidou, I. Gialampoudikis, D. Liparas, G. Casamayor, L. Wanner, N. Heise, et al. 2018. “A Multimodal Analytics Platform for Journalists Analyzing Large-Scale, Heterogenous Multilingual, and Multimedia Content.” Frontiers in Robotics and AI 5: 123. https://doi.org/10. 3389/frobt.2018.00123. Witt, L. 2017. “Preventing the Rogue bot Journalist: Protection from Non-Human Defamation.” Colorado Technology Law Journal 15 (2): 517–548. 20 M. SIITONEN ET AL.