Cycles - Blending Natural and Artificial Properties in a Generative Artwork

GA2010 – XIII Generative Art Conference – Politecnico di Milano University, Italy Daniel Bisig and Tatsuo Unemi Paper: Cycles - Blending Natural and Artificial Properties in a Generative Artwork Abstract: Cycles is an interactive installation that establishes an intimate relationship between the visitor's physical body and simulated organisms. It explores notions of transience and identity that draw inspiration from Buddhist philosophy. Cycles creates a situation that causes the visitor to experience his or her own body in a state of mutability and transience. Cycles merges the appearance of the visitor's hand with a visual representation of a swarm simulation. By bridging the gap between the virtual and physical, a hybrid entity comes into existence whose rapidly changing body blends artificial and natural properties. This hybrid entity progresses through a life cycle that reenacts the four Buddhist sufferings. The Visitor’s Hand as a Hybrid Entity Topic: Swarm & Dance Authors: Daniel Bisig Zurich University of the Arts, Institute of Computer Music and Sound Technology Switzerland www.icst.net Tatsuo Unemi Soka University, Department of Information Systems Science Japan www.intlab.soka.ac.jp/ ~unemi/ Contact: [email protected] [email protected] Keywords: interactive installation, body projection, swarm simulation, Buddhism page 140 13th Generative Art Conference GA2010 Cycles Blending Natural and Artificial Properties in a Generative Artwork Daniel Bisig Institute for Computer Music and Sound Technology Zurich University of the Arts e-mail: [email protected] Tatsuo Unemi Department of Information Systems Science Soka University e-mail: [email protected] Abstract Cycles is an interactive installation that establishes an intimate relationship between the visitor's physical body and simulated organisms. It explores notions of transience and identity that draw inspiration from Buddhist philosophy. Cycles creates a situation that causes the visitor to experience his or her own body in a state of mutability and transience. Cycles merges the appearance of the visitor's hand with a visual representation of a swarm simulation. By bridging the gap between the virtual and physical, a hybrid entity comes into existence whose rapidly changing body blends artificial and natural properties. This hybrid entity progresses through a life cycle that reenacts the four Buddhist sufferings. 1. Introduction Cycles employs an interactive generative system to create visuals that become perceivable on the surface of the visitor’s hand. It draws from conceptual ideas and technological means that have been developed throughout various projects in engineering and art that employ video projection on the human body. Artistic motivations for employing this technology usually center around a desire to transform the human body into something more malleable and ephemeral than it usually is and to blur the boundaries between the physical and the intangible. The flexibility and visual quality of body projection has increased alongside technical developments in video projection, video tracking and video processing. In particular, the combination of contemporary dance and new media has been and still is an important driving force to adapt and transfer these technological advances into the domain of the arts [1]. The application of body projection as a real time and interactive aspect in an artwork or stage performance has long been hampered by a lack of computational power. page 141 13th Generative Art Conference GA2010 Earlier dance performance works by Klaus Obermaier such as D.A.V.E (premiered in 1999) and Vivisektor (premiered in 2002) employ finished movies as visual content for body projection [2]. Similarly, the dance trilogy De Humani / Corporis / Fabrica by Pablo Ventura that was premiered in between 2002 and 2005 and that employs body projection as important choreographic element also relies entirely on pre-made movie material [3]. Accordingly, these performances required a perfect timing and positioning of the dancers’ movements in order to achieve the desired overlap between the dancers’ appearance and the video projection. With the advent of more powerful computers and graphics hardware, the possibility for live processing of camera images and real time generation of visual feedback freed the artists and dancers from the constraints of entirely predetermined body visuals and allowed for a much higher degree of spontaneity and personal expressivity than previously possible. More recent works such as Apparation by Klaus Obermaier (premiered in 2004) [4] or Mortal Engine by dance company Chunky Move (premiered in 2008) [5] emphasize this newly gained freedom in body projection by employing real-time generative visuals that are very responsive to the dancers’ movements. The authors of this article have also been involved in two dance projects that employ real time interactive visuals for body projection. Here, the visuals are generated via swarm simulations that react to the dancers’ movements [6]. The first performance entitled Gods and Dogs was choreographed by Ji í Kyliánř and premiered in 2008. The second performance entitled 2047 was choreographed by Pablo Ventura and premiered in 2009. Contrary to body projection in dance choreography, the application of this technique in installation art is very rare. One notable example is the work entitled Mutsugoto by DistanceLab [7]. This installation displays the movement of remote touch gestures as white traces on the bodies of the two interacting participants. Even rarer is video projection on only parts of the human body such as the human hand. Here, one of the most famous examples stems from a research project in human computer interaction (HCI) entitled Sixth Sense. This project employs a wearable camera and beamer setup to display visual information on physical objects. In one of the scenarios of this project, the human hand is transformed into an interface via video projection [8]. Another HCI project entitled Information Falls Like Rain also employs the hand as surface to display visual information [9]. But contrary to the Sixth Sense project, the setup consists of a stationary camera and beamer and allows several users to cooperate in accessing and manipulating information via their hands. The authors are aware of only one installation artwork that employs video projection on the visitors’ hands. This artwork by Chris Sugrue is entitled Delicate Boundaries [10]. It is of particular relevance here, since it not only shares the hand projection aspect with Cycles but also employs visualizations of virtual creatures that move over the visitor’s body. 2. Concept The project Cycles has emerged from the authors long standing interest into the relationships and roles that can be established between humans and simulated artificial organisms. Cycles experiments with an extreme form of relationship, one that fuses the human body with artificial organisms and thereby gives rise to a hybrid entity that blends natural and artificial properties. This fusion was achieved by projecting a visual rendering of the artificial organisms directly onto the visitor’s hands. The hand as a projection surface was chosen since it constitutes a prominent part of the human body. It is one of the few body parts we are almost always page 142 13th Generative Art Conference GA2010 consciously aware of. It is an integral component for our perception and action capabilities. It provides detailed tactile sensations, it is almost always present in our visual field and it is essential for active exploration and manipulation. For these reasons, we concluded that the transformation of the visual appearance of the visitor’s hands via body projection would have a strong effect on the visitor’s self observation and awareness. In order to explore various manifestations of such a close relationship between natural and artificial organisms, the properties and behaviors of the artificial organisms gradually change during a visitor’s interaction with the installation. During certain stages, the artificial organisms treat the visitor’s body as a host in either a symbiotic or parasitic relationship. During other stages, the artificial organisms become part of the visitors body by behaving as cells or cell groups. The visitor witnesses the occurrence of these changes while the hybrid entity progresses through a brief live cycle. The visitor experiences the birth of the hybrid entity at the beginning of the interaction, observes its growth and maturation and is finally confronted with its death. The design of the life cycle draws inspiration from Buddhist philosophy. According to Buddhist philosophy, the concept of self and the experience of identity undergoes a continuous process of development and change. Every attempt to cling to a fixed notion of identity results in the painful experience of the four sufferings of life: the suffering of birth, aging, sickness and death. Cycles revisits this notion of identity. It creates a situation that causes the visitor to experience his or her own body in a state of mutability and transience. The life cycle of the hybrid entity reenacts the four Buddhist sufferings. 3. Implementation 3.1 Hardware Cycles has been exhibited as a small installation that is placed in a dark room and that is intended to be experienced by a single visitor at a time. The hardware of the installation consists of a table or pedestal that is 105 cm high and that is painted in a black color that absorbs near infrared light (see Figure 1). The table houses two Mac Mini computers that operate under Mac OS X 10.6 and that run custom developed software for video tracking, simulation and visualization and that communicate with each other via an ethernet connection. Attached to the table is a custom support structure that vaguely resembles a desk lamp. It houses a small video projector, a camera whose lens is covered with a daylight blocking filter and an array of infrared leds that serve as light source (see Figure 2). The projector, camera and infrared light source are all directed towards the surface of the table. The daylight blocking filter prevents the camera from detecting the projected video image. The infrared light source provides sufficient light for the camera to detect the visitor’s hand as a bright object in front of a black background. page 143 13th Generative Art Conference GA2010 Figure 1: Cycles Installation. The photo depicts the table version of the installation. Figure 2: Cycles Installation Detail. The photo shows the structure that houses a video projector, a camera and an infrared light source. 3.2 Software The software component of the installation includes several different custom developed applications. 3.2.1 Video Tracking page 144 13th Generative Art Conference GA2010 A video tracking software combines standard computer vision functionality provided by the OpenCV library [11] with algorithms for geometric interpretation of hand contours. The software discriminates between hand-like and non-hand-like objects based on the number and depth of concave indentations in the contour of the detected objects. For hand-like objects, it determines the position, orientation and length of the fingers and wrist and the size and center of the palm (see Figure 3). The tracking software runs on one Mac Mini and sends the geometrical hand information via UDP/IP to the simulation software that runs on the other Mac Mini. 3.2.2 Agent Simulation The simulation and visualization software consists of three different programs. This segregation of the software was necessary because the two authors work with different development environments (Nokia Qt and Apple Cocoa) and in different programming languages (C++ and Objective-C). The three different programs correspond to different stages in the hybrid entity’s life cycle. The simulation programs implement a two dimensional swarm simulation that comprises not only BOIDS type of flocking behaviors [12] but also additional behaviors. Some of these additional behaviors allow the agents to respond to the presence and position of various hand properties whereas other behaviors cause the agents to change their visual appearance. Figure 3: Hand-Tracking Software. The screenshot depicts the hand’s contour as yellow lines, the fingers’ and wrist’s orientation as red lines, and the boundaries of the palm as gray lines. page 145 13th Generative Art Conference GA2010 Figure 4: Simulation Software. The screenshot depicts the chemicals and agents that respond to the hand’s presence during the stage of sickness The second and third simulation programs also implement artificial chemicals that are distributed within the agents’ environment and whose concentration are changed by the agents, the visitor’s hand, and the chemicals themselves. Since the agents also react to the presence of these chemicals, these simulations implement a simple form of stygmergic interaction among the agents and between the agents and the visitor (see Figure 4). The first and third simulation programs have been implemented based on software libraries that have been previously developed as part of a research project entitled Interactive Swarm Orchestra [13, 14]. The behaviors and appearances of the flocking agents and chemicals gradually change while each of the simulation programs is running. These changes are prechoreographed and occur over a fixed amount of time. Once a simulation program has come to an end, the next simulation program is automatically started. Once the third and last simulation program has quit, the life cycle begins again with the starting of the first simulation program. The entire duration of a life cycle is about 5 minutes. The first simulation program controls the stage of birth. Initially, the agents are scattered as small white dots across the table surface and remain mostly stationary (Figure 5a). This stage is maintained as long as no hand is detected. When a hand appears, the agents move towards the center of the palm in a spiraling motion (Figure 5b). In the center of the hand, the agents start to oscillate and assume cell like shapes (Figure 5c). Finally, they begin to follow the contour of the hand, tracing it with lines that represent their trajectories (Figure 5d). page 146 13th Generative Art Conference GA2010 Figure 5a: Stage of Birth, Phase 1 Figure 5b: Stage of Birth, Phase 2 Figure 5c: Stage of Birth, Phase 3 Figure 5d: Stage of Birth, Phase The second simulation program controls the stage of growth. This simulation employs a combination of swarm simulation and chemical simulation. There exist two type of chemicals that diffuse across a two dimension grid that spans the simulation space. The first chemical is produced by the visitor’s hand and exerts an attractive force on the agents. The second chemical is produced by the agents and causes the agents to avoid each other. At the beginning, agents appear as fast moving small dots representing frog spawn and are attracted to the visitor’s fingertips (Figure 6a). From there, the agents slowly move along the fingers towards the center of the hand. During this travel, the agents’ shapes transform from spawn to tadpole (Figure 6b) . Once the agents have reached the center of the hand, their shapes transform into that of a frog. In this form, the agents move in a jumpy motion towards the wrist of the hand (Figure 6c). page 147 13th Generative Art Conference GA2010 Figure 6a: Stage of Growth, Phase 1 Figure 6b: Stage of Growth, Phase 2 Figure 6c: Stage of Growth, Phase 3 Whenever the agents loose contact with the hand, they are immediately transformed back into their initial frog spawn phase. The third simulation controls the stages of sickness, aging and death. This simulation also employs a combination of swarm simulation and chemical simulation. The chemical simulation implements a Gray-Scott Reaction Diffusion System [15] and is implemented in OpenCL to benefit from the processing speed of the graphics hardware [16]. The stage of sickness starts with a random distribution of agents that appear as green dots and gradually elongate into bacteria shapes (Figure 7a). Subsequently, the hand starts to emit chemicals that appear as white and red blobs. The agents are attracted to these chemicals and remove them when they move across. (Figure 7b). page 148