The scenographic space(s) of augmented reality

327 Introduction There has been a recent wave of research attention focusing on immersive and mixed reality technologies across a range of artforms, including theatre and performance. Accompanying this is a pervading sense that the creative potential of these technologies has not yet been realised. Somewhat surprisingly, there has been very little work in this area within a specifically scenographic context. This paper will focus specifically on augmented reality (AR), drawing on frameworks within scenography to argue that this context provides a way to think through and experiment with the affective potential of these technologies. AR is not necessarily understood here as associated with a particular technology. Rather, I define it as any handheld or wearable technology that creates an augmented experience of space. Contrary to most existing research in AR, I resist thinking in terms of a binary distinction between virtual objects or information, and ‘reality’. I instead build on the work a small minority of scholars who argue that AR creates new mediated spaces or realities. I take this further, proposing that AR, like the related but distinct virtual reality, can be discussed in terms of its spatiality. I consider this spatiality in the context of a practice-research performance installation Ernest THE SCENOGRAPHIC SPACE(S) OF AUGMENTED REALITY LUCY THORNETT 328 Remains, staged at the University of Leeds in 2019. In this installation, audience members explored a multi-roomed space using a handheld tablet. I analyse the audience experience of this work through a discussion of audience responses provided during post-show interviews and reflection on insights that emerged during my own experiences of the installation. Drawing on existing theoretical frameworks within scenography, I contend that this scenographic experiment reveals the way in which embodied relations with AR produce affective, speculative space(s). Augmented Reality Immersive technologies, such as augmented reality, are not new, but recent developments in technology making them more accessible have sparked renewed interest in their potential by researchers and artists. In the case of augmented reality, the ubiquity of smartphone and their progressively more technologically sophisticated AR capabilities means that AR technologies are increasingly becoming embedded within the everyday (think, for example, of face filters on camera apps or GPS location data). AR is often associated with particular technologies that use the cameras of handheld devices such as smartphones or Head Mounted Displays to overlay virtual information or objects onto the ‘real’ world. It should be noted, however, that terms associated with mixed reality technologies are undergoing rapid change, often driven by tech industry marketing. Academic research in this area tends to define it more broadly. According to Ronald Azuma, the three key characteristics of AR are that it ‘combines real and virtual’; ‘is interactive in real time’; and ‘is registered in three dimensions’ (1997). Most research in AR follows Azuma’s lead in defining it according to its ability to blend virtual and physical, rather than in relation to a particular technology. Milgram and Kishino’s ‘Reality-Virtuality Continuum’ is also influential in defining Augmented and Mixed Reality (1994). They place AR on a Mixed Reality scale that places the ‘real environment’ at one end and the ‘virtual environment’ on LUCY THORNETT 329 the other. However, this continuum has been critiqued for its linearity and the binary it constructs between the real and the virtual (see, for example, Benford and Giannachi, 2011). In an artistic context, researchers have tended to define AR quite broadly – a number have cited Janet Cardiff’s audio walks as paradigmatic examples (e.g. Manovich, 2006; Wright, 2018). An early Cardiff work, The Missing Voice (1999) guided audiences through an area of London using sound recorded in the same location (relating to a fictional narrative), played back through headphones. Later examples such as City of Forking Paths (2014) have used sound combined with videos displayed on a smartphone. These works, understood as AR, demonstrate the breadth of possibilities for overlaying virtual space onto a physical location, unmoored from a specific technology. I propose that AR differs from other mixed reality technologies – such as, for example, projection – in that it augments space from the position of the body, through either headworn or handheld technology1. Research in AR crosses a wide range of disciplines and applications, but has been less well explored within artistic contexts. Scholarly practice in AR art to date has broadly focused on surveying the field (Geroimenko, 2018) or outlining its political potential as a tool for activism (e.g. Skwarek, 2018; Thiel, 2018)2. However, of relevance here is a small but growing body of research discussing the embodied relations produced by AR art. Patrick Lichty discusses embodied encounters with both handheld and head-mounted AR artworks. He argues that AR produces ‘a line of attention/flight between the interactor and the superimposed media overlaid on the given environment’ (2018, p.137), which he calls a ‘performative gestural gaze’ (p.134). Lichty’s work is emblematic of much research in AR in that it takes literally AR’s ostensible premise of augmenting 1 This differs somewhat from other broad definitions particularly within a theatre and performance context. For example, Benford and Giannachi (2011) use the term ‘mixed reality’ while Vincs et al (2018) use augmented reality – both include projection technologies within their definitions. 2 These examples contain an interesting discussion on AR’s ability to intervene into spaces of power due to its ability to hide things in plain sight. THE SCENOGRAPHIC SPACE(S) OF AUGMENTED REALITY 330 ‘reality’ with virtual objects or information. This, according to the dominant paradigm in AR, differentiates it from VR – as rather than being immersed in a virtual space, the user’s primary experience is of the real physical space. This again underscores the way in which much scholarship in VR and AR sets up a problematic dichotomy between virtuality and reality. I contend that this binary fails to account for AR’s ability to enact speculative worlds, in a manner similar, but distinct from VR. A few researchers have discussed the way that AR might produce an alternate space or reality. Borko Furht argues that AR ‘modifies the spatial configuration of reality’ (2011, p.61) and produces a ‘new environment’ (p.48). Rewa Wright also contends that AR posits a ‘parallel reality’ that arises out of the relationality of body and artwork (2014), though for her this reality is distinct from, but connected to, physical space. Furht’s and Wrights framings point to multiple ‘realities’ or orders of space, rather than a binary distinction between reality and virtuality. Following this, we can begin to open AR to a discussion of spatiality – expanding its attendant questions of embodiment to a discussion of bodily immersion across multiple registers of space. I propose that this offers a distinctly scenographic register with which to experiment with AR. Existing research in scenography provides some frameworks to think through scenography’s role in crafting embodied relations in and with space(s). Joslin McKinney and Scott Palmer define scenography as ‘a mode of encounter founded on spatial and material relations between bodies, objects and environments’ (2017, p.2). This notion of scenography as a relational practice is also echoed by other researchers (e.g. Aronson, 2017; Hann, 2019) McKinney and Palmer identify relationality as a key concept or understanding how scenography operates in and beyond performance. Taking AR technology as an interface between bodies and various virtual and physical registers of space, it can also be seen to function relationally. This highlights one way we can think about the scenographic and spatial potential of AR. However, in proposing that AR enacts speculative worlds, I contend that we can go further in discussing how a LUCY THORNETT 331 scenographic framing might bring to light AR’s spatiality, and in doing so, its affectivity. Ernest Remains Ernest Remains was a practice-led performance installation staged at the University of Leeds in 2019. It was designed as short experience of approximately twenty minutes, for one audience member at a time. Audiences were given a set of headphones and a handheld tablet and entered a multi-roomed installation that utilised the entire space of the theatre, including dressing room and auditorium. Footage of a figure moving around the space of the installation appeared onscreen periodically, guiding the audience through the environment. In between these interludes, the tablet could be held over objects in the space to trigger sound and video cues as well as virtual objects appearing on the screen. The content of the piece was autobiographical and related to my grandfather, Ernest Thornett, who was a detective fiction author and code-breaker for British intelligence during the second world war. The piece drew on ‘drawing-room mystery’ tropes and cast the audience member as detective, using the tablet to discover clues. The space was filled with archival material relating to Ernest, such as letters and photographs, as well as contemporary anachronistic material and objects such as print-outs of emails and online chats, digital screens, and floorplans of the installation space. Thematically the work explored the digital and material traces bodies leave behind, and touched on contemporary and historical notions of surveillance. In doing so these themes intentionally overlapped with the research aims of investigating the relations between bodies, matter, environments and technologies. Audiences also wore a go-pro camera that recorded what they saw, and this footage was used to prompt discussion during post-show interviews. I will discuss this work through an analysis of my own experience, and comments made by audiences. Where quoting directly from interviews, I have used my own initials (LT) to indicate myself as interviewer, and the letters A-G to indicate different members of the audience. Two key aspects of the performance THE SCENOGRAPHIC SPACE(S) OF AUGMENTED REALITY 332 emerged where the AR produced a scenographic spatiality, and I will focus on these moments in my discussion here. The first of these was the experience of ‘following’ the filmed performer through the space. For many audience members, this created the sensation that the performer inhabited virtual and physical space simultaneously: A: I liked that sensation of following somebody. When I felt comfortable, when I knew where I was going. LT: What did you like about it? A: That it wasn’t real, that he wasn’t there. But he kind of was. That was the…almost felt more theatrical than the other bits and pieces. It felt like there was this real person who wasn’t there. LUCY THORNETT Figure 1. Ernest Remains augmented reality installation at Banham Theatre, University of Leeds (2019) by author. An audience member viewing the installation through the screen 333 The use of the word ‘theatrical’ here is interesting in relation to this slippage between virtuality and actuality. Antonin Artaud described theatre as a ‘virtual reality’, discussing the way that it posits an ‘illusory world’ (1958, p.49). A number of scholars have pointed out that theatre is inherently concerned with virtuality (e.g. Popat, 2016; Thomas and Glowacki, 2018). For Brian Massumi, the virtual is not illusory, but ontologically real. He defines the virtual as the ‘“real but abstract” incorporeality of the body’ (2002, p.21). For Massumi, the term virtual here is not synonymous with digital realities, however others including Mark Grimshaw have argued that digital technology might provide us with new ways of accessing this virtuality (2014). Massumi’s work is useful here in understanding that the virtual is no less real than physical reality. Instead it consists of a different materiality, one that can still be felt in the body. In this sense, we can relate virtual reality to theatrical reality in that though it is speculative, it is simultaneously ontologically real, and perceived as such through THE SCENOGRAPHIC SPACE(S) OF AUGMENTED REALITY Figure 2. Ernest Remains augmented reality installation at Banham Theatre, University of Leeds (2019) by author. Audience following a filmed performer through the space 334 the sensing body of the audience. I will turn now to another moment in the performance to illustrate this further. In the final part of the performance, audiences were guided to return to the space they had started in. While they were elsewhere in the space, the curtains had opened to reveal posters of trees on the auditorium floor. A fan had been turned on, creating a gentle breeze that moved the curtains. When audiences directed the tablet at the trees, life-sized virtual trees appeared that remained in place if the screen was moved up towards the leaves at the top. For some audiences, the trees registered as a solid presence in the space. For these participants there was an embodied instinct to avoid walking through the trees, despite knowing that they weren’t physically there, as seen in the comments from two audience interviews below: B: Yeah the trees were, that was a very cool feeling, sort of walking in and… knowing where I could move and where I could walk suddenly became a bit of like a ‘Oh, wait’. I almost questioned whether I could walk over the tree… Yeah, it felt definitely felt like physical obstacles and I was sort of almost… moving around them rather than sort of confidently moving through them. LT: Oh, you moved your body to get around them? C: Yeah, I went like that to try and get round it. And I was like (imitates movement) ...that’s ridiculous. LT: That’s quite interesting. So then there’s a kind of bodily feeling that the tree is in the space? C: Yeah. Even though I knew it wasn’t. The virtual trees here are visually seen on-screen, but felt to be co-present in the same physical space as the audience – they have a virtual materiality. The fact that this slippage of virtual into actual was described by the participant above as producing a ‘cool feeling’ points to the affective nature of this experience. For LUCY THORNETT 335 Massumi, affect is temporal – a ‘state of suspense’ (1995, p.86) or pre-conscious intensity that allows the virtual to enter, briefly, into our conscious experience. In fact, he argues that it can only be understood temporally and not spatially (2011, p.16). However, an atmospheric and scenographic perspective might reveal this affective experience of the virtual to be spatial as well as temporal. There is a growing body of scholarship on the spatiality of these affective processes. Derek P. McCormack uses the term ‘affective spacetimes’ to articulate the inseparability of spatiality and temporality in this process: Space, in other words, is never a backdrop for something more dynamic. Nor indeed, can it or should it be juxtaposed to process or temporality in a way that privileges the latter. Instead, it is always more accurate to speak of space, or spaces, as multiple: spaces produced via a range of technologies and experienced through different sensory registers; spaces with variable reaches and intensities; and spaces that can often only be apprehended in and through the assemblages of movement and stillness of which they are composed. (2014, p.2) This discussion of spaces as multiple, as produced by bodies in movement and through technologies underscores the way that the AR technology in Ernest Remains enacted speculative scenographic space(s). Kathleen Stewart similarly speaks of atmospheric ‘attunements’ – ‘an attention to the… complex emergent worlds…’, which, following Heidegger, she calls ‘worlding’ (2011, p.445). Like McCormack, she discusses the embodied practices through which such worldings emerge (p.446). This helps to think through the way in Ernest Remains, the moving body in concert with handheld screen, digital content and headphones – produced a particular spacetime or atmosphere. There are a number of ways that this spatiality was seen to manifest in the performance. One of these was a particular directionality necessitated by the THE SCENOGRAPHIC SPACE(S) OF AUGMENTED REALITY