Augmented Reality and Journalism: 10 use-case analysis from television, printing and web media platforms

Recibido: 00/00/2020 Aceptado: 00/00/2020 Realidad aumentada y periodismo: análisis de 10 estudios de caso en televisión, prensa y plataformas digitales Augmented Reality and Journalism: 10 use-case analysis from television, printing and web media platforms Abstract Augmented reality has inaugurated new developments that enable the creation of content of various kinds. Based on the case study methodology, the work analyzes 10 successful experiences of using augmented reality linked to television, the press and digital media. The investigation concludes that the use of augmented reality without a journalistic purpose can harm the media. Resumen La realidad aumentada ha inaugurado nuevos desarrollos que posibilitan la creación de contenidos de diversa índole. A partir de la metodología del estudio de caso, el trabajo analiza 10 experiencias de éxito de uso de la realidad aumentada vinculadas con la televisión, la prensa y los cibermedios. La investigación concluye que el uso de la realidad aumentada sin una finalidad periodística puede perjudicar a los medios. Keywords Augmented reality, immersive journalism, digital journalism, internet Recibido: 30/05/2020 Aceptado: 20/11/2020 Santiago Tejedor Calvo (Universidad Autónoma de Barcelona) [[email protected]] Natalia Cardona (Universidad Autónoma de Barcelona) [[email protected]] Laura Cervi (Universidad Autónoma de Barcelona) [[email protected]] DOI: http://dx.doi.org/10.12795/IC.2020.i01.19 E-ISSN: 2173-1071 IC – Revista Científica de Información y Comunicación 2020, 17, pp. 437 - 455 Santiago Tejedor Calvo / Natalia Cardona / Laura Cervi 438 IC – Revista Científica de Información y Comunicación 17 (2020) [pp. 437-455] E-ISSN: 2173-1071 Palabras clave Realidad aumentada, periodismo inmersivo, periodismo digital, internet Summary: 1. Introduction: Augmented Reality applied to Journalism 2. A look at technology: Augmented Reality and Journalism 3. Augmented Reality applied to digital journalism 4. Method 5. Data collection instruments 6. Cases studies 6.1. Television use cases 6.2. Printed magazines use cases 6.3. Web platform use cases 7. Conclusions 8. Bibliography Sumario 1. Introducción: la realidad aumentada aplicada al periodismo 2. Una mirada a la tecnología: realidad aumentada y periodismo 3. Realidad aumentada aplicada al periodismo digital 4. Método 5. Instrumentos de recogida de datos 6. Estudios de casos 6.1. Casos de uso de televisión 6.2. Casos de uso de revistas impresas 6.3. Casos de uso de plataforma web 7. Conclusiones 8. Bibliografía PERSPECTIVAS Augmented Reality and Journalism 439 E-ISSN: 2173-1071 IC – Revista Científica de Información y Comunicación 17 (2020) [pp. 437-455] 1. Introduction: Augmented Reality applied to Journalism With the intention of innovating, media companies are constantly searching for professionals, software, or equipment that can compete in speed and quality with the information that moves on the internet. By enriching journalistic publications with elements of Augmented Reality (AR), they seek to create greater interaction with the audience, the content and improve the perception of journalistic topics. According to Amy Webb, in the 2019 Tech Trends Report document written for Future Today Institute, “mixed reality, also known as extended reality, is the field of digitally generated, enhanced or manipulated environments that encompasses Virtual Reality and Augmented Reality experienced through a device used in the head (Head-Mounted Device) or through mobile devices, mixed reality has become increasingly present in contemporary consciousness over the past decade. In 2019 we will see the mixed reality more deeply intertwined in various industries and in new commercially viable applications.”(Webb, 2019) The concept of Augmented Reality refers to the superposition, in real time, of images, markers or information generated virtually on real-world images. Physical reality is perceived by the five senses in the human being: hearing, smell, taste, vision and touch. Augmented Reality aims to complement this sensory perception of the world with digital elements that we can consume through our technological devices (Telefónica, 2011). This tool allows to create new media by making significant connections between virtual content and the surrounding real environment, generating experiences where the power comes from the connection and not only from the virtual or real world (Azuma, 2017b). The characteristics described are of value when it comes to generating journalistic content, so in this work we try to explore the possibilities that Augmented Reality has when it comes to informing in an increasingly digital world. According to Parra (2017), This combination of the real and the virtual is what differentiates it from virtual reality, which focuses exclusively on a non-real environment despite managing real-looking objects and scenarios in which the user has the feeling of being immersed in through one or several external devices such as glasses, helmets, gloves and even special suits. To Alex Kipman, creator of the Kinect and HoloLens, Augmented Reality has its greatest potential in communication “I believe that communication essentially defines a secular trend in computer science. It happens, that sooner it is not innovation in communication that transforms things. From slow mail to real mail, from there to messages, Santiago Tejedor Calvo / Natalia Cardona / Laura Cervi 440 IC – Revista Científica de Información y Comunicación 17 (2020) [pp. 437-455] E-ISSN: 2173-1071 text messages, to Snapchat. It’s like going from video to teleportation.” (Stein and Sherr, 2019). For Azuma (2017b), Augmented Reality has greater long-term potential than other types of mixed realities because it allows the user to interact and understand the real world better. This allows users to connect with people, places and nearby objects instead of isolating them from the surrounding environment. As Parra (2017) describes, in 1998 the first edition of the IWAR (International Workshop for Augmented Reality) is held, which brings together researchers and scientists, technology developers, device manufacturers and suppliers of products and services from around the world. After two editions (2000 and 2001) in which it was known by the acronym ISAR (International Symposium on Augmented Reality), from 2002 it was renamed ISMAR (International Symposium on Mixed and Augmented Reality) and began to integrate, in addition to the sectors previously indicated, to telecommunications operators, vertical application developers, content providers and representatives of the artistic world. This system has the potential to replace the screens by popularizing the use of wearable devices, however, there are still technical and experiential advances that allow this change. The development of attractive experiences to solve image, optical and visualization problems that improve the use of this technology in different fields of application should be encouraged. (Azuma, 2017b). Given the above, the possibility of starting to create journalistic content using this technology would be of great contribution when it comes to improving and strengthening the use of Augmented Reality in all areas. 2. A look at technology: Augmented Reality and Journalism According to the taxonomy presented by the engineers Milgram and Kishino (1994), the concept of “virtual continuum” refers to the mixture of classes of objects presented in an exhibition situation, where real environments are shown at one end of the continuum and virtual environments are at the opposite end. See Figure 1. In accordance with what is proposed by the authors at the corresponding end called the real environment is the content that is composed only of real objects, even when they are shown through a screen. In the opposite side there are environments that consist solely of virtual objects, for example, computer generated simulation graphics. According to Figure 1, proposed by Milgram and Kishino (1994), the way to understand Mixed Reality environments is to take into account that they are those in which PERSPECTIVAS Augmented Reality and Journalism 441 E-ISSN: 2173-1071 IC – Revista Científica de Información y Comunicación 17 (2020) [pp. 437-455] real environments and virtual environments are presented together, in a single device, at any point of the virtual continuum. According to the proposed taxonomy there are several kinds of existing hybrid deployments. The environments that can be considered as Mixed Reality interfaces are: 1. “Window-on-the-world” (WoW) display, in which computer-generated images are superimposed electronically or digitally and viewed through a monitor. An example of this would be the chroma-key. 2. There is a visualization of computer -generated images superimposed electronically or digitally, but they are observed on monitors or immersive devices used in the head - head mounted displays (HMD’s) -. 3. Monitors or immersive devices used in the head (HMD’s) with the ability to see through them, in which computer-generated graphics can be superimposed, using mirrors, on real-world scenes that can be seen directly. 4. Immersive monitors used on the head with the ability to see through them, but video is used, instead of optical, observing the outside world. The difference between class 2 and class 4 is that with class 4 the world must be seen VIRTUALITY CONTINUUM (VC) REAL ENVIRONMENT AUGMENTED REALITY (AR) AUGMENTED VIRTUALITY (AV) VIRTUAL ENVIRONMENT MIXED REALITY (MR) FIGURE 1 Simplified representation of the virtual continuum. Source: Milgram and Kishino, 1994 Santiago Tejedor Calvo / Natalia Cardona / Laura Cervi 442 IC – Revista Científica de Información y Comunicación 17 (2020) [pp. 437-455] E-ISSN: 2173-1071 orthotically (correct proportions) with the immediate real world, creating a transparent video system. Analogous to the optical display of class 3. 5. Fully graphic viewing environments, whether fully immersive, partially immersive or otherwise, to which real video is added. 6. Fully graphic, but partially immersive environments in which real physical objects in the user’s environment play a role in - or interfere with the computer generated scene. For example, reach and grab a digital object with your own hand. Considering the above classification, we can then say that the concept of Augmented Reality refers to any case in which a real environment is complemented by means of computer-generated virtual objects. (Milgram and Kishino, 1994). For Azuma (1997a), the concept of Augmented Reality is defined from the following conception “to avoid limiting Augmented Reality to specific technologies, this article defines AR as systems that have the following three properties: 1) It combines the real and the virtual. 2) Interactive and in real time. 3) Registered in 3D”. In the article A survey of Augmented Reality, published by Ronald Azuma (1997a) the basic characteristics of Augmented Reality systems are described. Focused on creating increased content through optical or video technology. The characteristics stated by the author are: 1. Augmentation: Augmented Reality has the potential to add digital objects or hide real user objects, it has focused on adding objects to real environments, although objects can also be removed, as has been done for example in film productions. It can be applied to all senses; it must not necessarily be focused on sight. It can be extended to include sound by means of audio devices that combine 3D synthetic sound with real sounds taken from the real environment, this could allow selectively canceling sounds that come from the user’s environment. 2. Optic or video: When developing an AR system, the elements to be used must be considered to achieve the objective of combining real and virtual elements. To achieve this effect, optical or video technologies are used. The immersive monitor devices used in the head with the ability to see through them are the option chosen for the sample and consumption of RA contents because they allow user interaction with the real world by superimposing virtual objects. PERSPECTIVAS Augmented Reality and Journalism 443 E-ISSN: 2173-1071 IC – Revista Científica de Información y Comunicación 17 (2020) [pp. 437-455] 3. Focus and contrast: The focus and contrast of objects created for RA present several challenges. In order to have a good focus on the objects it is necessary that the computer that processes the information takes into account the depth of field. In the case the contrast must be taken into account that the human eye has a high capacity to differentiate and adapt to different ranges of light, so that the vision of the real and virtual world must be compressed in the dynamic range of the device through which it is accessed at the information. 4. Portability: For Augmented Reality applications to work in an integrated manner with the environment, they must maintain the information taking into account that the user can move through wide spaces, in uncontrolled environments. For the correct operation of Augmented Reality systems, three basic subsystems are required. First, there is the scene generator, which provides the virtual objects or information. Then, display device, where the contents will be consumed. Finally, monitoring and detection, to access the contents from the points where an activator is required. According to Prendes (2015), levels can be understood to measure the complexity of the technologies involved in the development of RA systems, at a higher level the possibilities of application increase. Level 0: Physical world hyper linking. Use barcodes, 2D codes (such as QR), or image recognition. At this level the codes are hyperlinks to other content, there is no 3D registration or bookmark tracking. Level 1: Marker based AR. It is configured from the recognition of 2D patterns or recognition of 3D objects. Level 2: Markerless AR. Its operation is based through the use of GPS, compass or accelerometer of electronic devices in a way that determines the situation and orientation and superimpose the points of interest with the images of the real world. Level 3: Augmented vision. The moment the RA becomes an augmented vision, it becomes an immersive experience. Becoming something more relevant, contextual and personal. Santiago Tejedor Calvo / Natalia Cardona / Laura Cervi 444 IC – Revista Científica de Información y Comunicación 17 (2020) [pp. 437-455] E-ISSN: 2173-1071 Level 04: As quoted in Prendes (2015), Rice talks about a Level 4 where we will end up using contact lens displays and/or direct interfaces to the optic nerve and the brain. At this point, multiple realities will collide, mix and end with Matrix. When talking about current journalism we observe the multiplicity of opinions and theoretical aspects that seek to define the interaction and changes in the practice of the trade since the arrival of the internet and the technologies that accompany it. For Tejedor (2006), the first time a computer is used in journalistic writing is done in 1963, the Daily Oklahoma uses the IBM 1620 computer to compose its March 5 publication. From this moment the computer gains prominence progressively until it becomes essential today. The author describes the digitalization of the layout process from the use of a computer as a first step towards the journalistic practice we know today, but it was the arrival of the Internet that caused the greatest changes in the dynamics of the trade. Printing and distribution jobs disappear, dissemination is done through telematic networks and content is consumed through screens. Journalism undergoes a fundamental transformation and must rethink its processes in the new digital era. For Larrondo (2016), the journalistic narrative seeks to approach the creation of transmedia content, implying the need for journalism to experiment in conception of content for the web. This process occurs in the context of the digitalization of the media and the emergence of cyber journalism. As a storytelling tool, journalism is continually encouraged to apply changes in the way of telling and consuming stories, although it may be asked whether these developments are as such and if they manage to modify not only the form, but also in the background, the narrative practices already known, justifying the use of new labels (Domínguez, 2012). As Carmen Peñafiel (2016) states, the digital elements have integrated the essential elements of online communication: hypertextual content, audiovisual tools, the paradigm of immediacy from mobile media to learn about real-time information and social networks that allow sharing and disseminating information efficiently. From there, everything translates into a combination of elements to enrich the contents and feedback the communication. On the other hand, current journalism also stands out for its speed and instantaneity in the dissemination of messages. For López and Pereira (2010), the narrative and content convergence, facilitated by technology, has been supported within the communicative companies by various processes of parallel confluence in the strategic and professional scope. When looking to the future, the authors (López and Pereira, 2016) refer to the use of screenless technologies as the next step in the evolution of technological devices, which PERSPECTIVAS Augmented Reality and Journalism 445 E-ISSN: 2173-1071 IC – Revista Científica de Información y Comunicación 17 (2020) [pp. 437-455] relates to the obligation of journalists to rethink content to adapt to these New formats of content consumption. Journalism should aim to create content in the following years with personalization, updating, adaptation to audience segmentation, exploitation of the potential of mobile technologies and the search for new formats (Peñafiel, 2016). For Machado and Gadini (2018), current journalism must focus on content innovation, so professional profiles must adapt to content creation taking into account the predominant digital ecosystem today. 3. Augmented Reality applied to digital journalism Digital journalism is a commitment to the future that responds to the need of readers, listeners, viewers and users to receive information from different media and in different ways. The demand for audiovisual content is growing, a perspective that is still valid because the number of Internet users increases every day, as does the consumption of videos and audios through the digital media that are still on the rise; It is increasingly common to hear about the news through formats that record in real time or immediately the events such as Streaming, Facebook Live, Twitter or any other application or social network. For Martinchuk (2010), it is not a passing fad, but a new way of working, where genders have lost their borders to create a single distribution center as a result of new technologies, the Internet and the convergence of media. The way the journalist works is changing. The media digitize processes, workflows, and publishing systems are replaced by content managers that separate information from the format in which it is presented. They can be accessed through the Internet and, therefore, from anywhere in the world. We exercise the power of instant and globalized communication and a language, visual language, which exceeds both the written and the spoken word. This visual language has no borders. It does not need translation. It is not subject to corporate support. It is one of the most powerful tools of our time as stated by Gentile (2015). 4. Method After a deep literature review on possibilities of use of Augmented Reality for journalism, the study uses a methodological approach based on case study of ten cases, carried out from an intentional non-probabilistic sampling technique (Otzen & Manterola, Santiago Tejedor Calvo / Natalia Cardona / Laura Cervi 452 IC – Revista Científica de Información y Comunicación 17 (2020) [pp. 437-455] E-ISSN: 2173-1071 In other cases such as W Magazine, there are advances in the relationship between the consumer and the content, by making interactive pieces - overlapping videos in which you choose the order of visualization - it begins to give the viewer the ability to create their own content consumption. For the cases of Winter Olympics of the New York Times and Cassini Spacecraft of Quartz a significant advance is made in user interaction by superimposing the digital contents in real environments from 3D models. Which finally brings us to the last case, The Enemy by Karim Ben Khelifa as an experience that increases the level of interactivity from the content in interview format created for the user. Thus, we observe that as the popularity of implementation of Augmented Reality in the media grows the content creation for other consumption devices grows, and will prospectively follow the path of other previously adopted technologies. 7. Conclusions Despite the actual high costs of creating experiences in Augmented Reality the experts in the field do not see this obstacle as major for the development of this type of content. The different platforms adopt it in spite of this. This is an enterprise that involves multiple areas of knowledge and a technical effort for which it is not yet developed in the field of journalism. The uses given to Augmented Reality experiences on media platforms are currently marked by the need to innovate, which could lead to a misconception about the potential of this technology. It was found that for cases in which the written concepts are abstract, the visual representation of superimposed digital objects in the real world is a tool useful that brings depth and context to words. Augmented Reality should be a meeting point between technology and reality, superimposing digital information across devices opens a range of possibilities in information, entertainment and art, as found from the analysed cases. There is no single way to create this augmented information, as the different projects show a range of content varying from video and photography, to 2D and 3D objects. It is necessary to highlight that, although there is not a single format for the creation of this content, the observed trend runs towards 3D representation of information displayed with the aim of bringing reality closer through digital content. From the analyses is possible to conclude that there is a need to conceive the Augmented Reality contents as tools that allow the users access to information that PERSPECTIVAS Augmented Reality and Journalism 453 E-ISSN: 2173-1071 IC – Revista Científica de Información y Comunicación 17 (2020) [pp. 437-455] otherwise wouldn’t be possible to experiment. In addition, to visually represent objects or information will be further enriched if it’s provided with interactivity. The ability of users to interact with virtual objects and their potential popularity is clearly demonstrated in the experiences that combine information curated by journalists with freedom to decide how and when to navigate content and create a linked path between them. It is important to keep in mind that in order to develop this type of experience content creators must be trained in the implementation of the technology and create work teams that can support the different stages from idea to publication. 8. Bibliography ––––––– Azuma, RT (1997a) A Survey of Augmented Reality. Presence: Virtual and Augmented Reality 6 (4): 355 - 385. ---- (2017b) Making Augmented Reality a Reality. Applied Industrial Optics: Spectroscopy, Imaging and Metrology (paper JTu1F.1):1-3. ––––––– Branch, J (2018). Augmented Reality: Four of the Best Olympians, as You’ve Never Seen Them. The New York Times. Available at: https://www.nytimes.com/ interactive/2018/02/05/sports/olympics/ar-augmentedreality-olympic-athletes- ul.html (accessed 12 March 2019). ––––––– Curcurito, D (2009) Behind the scenes of Augmented Esquire. Esquire Available at: https://www.esquire.com/news-politics/news/g371/augmentedreality-technology -110 909/?slide=1 (accessed 13 March 2019). ––––––– Domínguez, E (2012) Medios de comunicación masiva. México: Red tercer milenio. ––––––– Editorial. TIME’s ‘Optimist’ Issue in Augmented Reality. TIME Magazine 4 January, 18. ––––––– Editorial. TVE, Antena 3 y TV Canaria confían en Brainstorm para la noche electoral (2015) Panorama Audiovisual. Available at: https://www.panoramaaudiovisual. com/2015/05/25/tve-antena-3-y-tv-canariaconfian-en-brainstorm-para-la- noche-electoral/ (accessed 25 march 2019). Santiago Tejedor Calvo / Natalia Cardona / Laura Cervi 454 IC – Revista Científica de Información y Comunicación 17 (2020) [pp. 437-455] E-ISSN: 2173-1071 ––––––– Feldman, B (2019) The Best Use of Augmented Reality Right Now Is the Weather Channel’s. Available at: http://nymag.com/intelligencer/2019/01/the-weather- channels-augmented-realitysegments.html (accessed 20 March 2019). ––––––– Gentile, Bill (2015) Fotógrafo / Periodista Bill Gentile en Nicaragua. In: U.S Embassy Managua. Available at: https://ni.usembassy.gov/es/fotografo-periodista-bill- gentile-en-nicaragua/ (accessed 20 march 2019). ––––––– Larrondo Ureta, A and Teixeira, J (2016) La convergencia narrativa en el periodismo móvil. Aproximación a la integración del contenido audiovisual en los productos nativos para iPad, Estudios sobre el Mensaje Periodístico 22 (2): 777-792. ––––––– López, X and Pereira, X (2010) Convergencia digital. Reconfiguración de los Medios de Comunicación en España. Spain: Universidad Santiago de Compostela. ––––––– Machado, E and Gadini, SL (2018) La innovación como eje estratégico en el desarrollo periodístico. Pauta Geral-Estudos em Jornalismo 5 (2): 271-280. ––––––– Martinchuk, E (2010) Periodismo Mochilero. Available at: https://www.pagina12.com. ar/diario/laventana/26-157842-2010-12-01.html (accessed 18 February 2019). ––––––– Milgram, P and Kishino, F (1994) A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems 77 (12): 1321-1329. ––––––– Otzen, T and Manterola, C (2017) Técnicas de Muestreo sobre una Población a Estudio. International Journal of Morphology, 35 (1): 227-232. ––––––– Parra Valcarce, D, Edo Bolós, C and Marcos Recio, J C (2017) Analysis of the application of augmented reality technologies in Spanish mass media productive processes. Revista Latina de Comunicación Social 72 (16): 1670 -1688. ––––––– Peñafiel Saiz, C (2016) Reinventing journalism in the digital ecosystem and transmedia storytelling. adComunica: revista científica de estrategias, tendencias e innovación en comunicación (12): 163. ––––––– Prendes Espinosa, C (2015) Realidad aumentada y educación: análisis de experiencias prácticas. Pixel-Bit: Revista de Medios y Educación (46): 187-203. ––––––– Reyes, T (1999) Métodos cualitativos de investigación: los grupos focales y el estudio de caso. Fórum empresarial 4 (1): 74-87. PERSPECTIVAS Augmented Reality and Journalism 455 E-ISSN: 2173-1071 IC – Revista Científica de Información y Comunicación 17 (2020) [pp. 437-455] ––––––– Simons, Helen (2011). El estudio de caso: Teoría y práctica. Spain: Ediciones Morata. ––––––– Stein, S and Sherr, Ian (2019) Why AR is going to give you ‘Superpowers’ in the future. In: CNET. Available at: https://www.cnet.com/news/the-future-of-ar- according-to-microsoft ––––––– Telefónica, F (2011) Realidad Aumentada: una nueva lente para ver el mundo. Spain: Fundación Telefónica. ––––––– Tejedor, S (2006) La enseñanza del ciberperiodismo en las licenciaturas de periodismo en España. PhD Thesis, Universitat Autònoma de Barcelona, Spain. ––––––– USDOD - U.S. Department of Defense (1999) Development of interactive multimedia instruction. ––––––– Webb, A (2019). Future Today Institute Tech Trends Report 2019. Report Future Today Institute, US, January.