Handheld Augmented Reality in education

Handheld Augmented Reality in education Universidad Politécnica de Valencia DSIC Master thesis Inteligencia Artificial, Reconocimiento de Formas e Imagen Digital Santiago González Gancedo Supervised by: M. Carmen Juan Lizandra July 2012 This thesis has been partially supported by the Spanish APRENDRA project (TIN2009-14319-C02-01), and by the Ministry of Science and Innovation (MICINN) of Spain (grant BES-2010-035118). To Adriana Contents 1 Introduction 3 1.1 Motivation ....................................... 3 1.2 Scientific goals and research hypotheses ....................... 5 1.3 Thesis structure .................................... 6 2 State of the art 7 2.1 Introduction to Augmented Reality .......................... 7 2.2 AR in education .................................... 8 2.3 MRLE ......................................... 10 2.4 Tactile and tangible interfaces in AR ......................... 11 3Study 1: Mixed Reality Learning Environments 13 3.1 Introduction ...................................... 13 3.2 The game ....................................... 14 3.2.1 Game design ................................. 14 3.2.2 Description of the game ........................... 17 3.3 Materials and methods ................................ 19 3.3.1 Hardware ................................... 19 3.3.2 Software ................................... 22 3.4 Design of the evaluations ............................... 22 3.4.1 Participants .................................. 22 3.4.2 Measurements ................................ 22 3.4.3 Procedure ................................... 24 3.5 Results ......................................... 24 3.5.1 Learning Outcomes .............................. 24 3.5.2 Satisfaction Outcomes ............................ 26 3.5.3 Interaction .................................. 28 3.6 Conclusions ...................................... 29 4Study 2: Impact of handheld devices in AR for children 31 4.1 Introduction ...................................... 31 4.2 Design of the evaluations ............................... 32 4.2.1 Participants .................................. 32 4.2.2 Measurements ................................ 32 4.2.3 Procedure ................................... 32 4.3 Results ......................................... 34 v 4.4 Conclusions ...................................... 36 5Study 3: Tactile and tangible interfaces 39 5.1 Introduction ...................................... 39 5.2 Experiment ...................................... 40 5.2.1 Description of the experiment ........................ 40 5.2.2 Apparatus ................................... 41 5.3 Design of the evaluations ............................... 42 5.3.1 Participants .................................. 42 5.3.2 Measurements ................................ 42 5.3.3 Procedure ................................... 43 5.4 Results ......................................... 44 5.4.1 Time analysis ................................. 44 5.4.2 Satisfaction .................................. 46 5.4.3 Interaction .................................. 47 5.5 Conclusions ...................................... 48 6 Concluding remarks 51 6.1 Conclusions ...................................... 51 6.2 Scientific contributions ................................ 52 6.3 Future work ...................................... 52 Bibliography 54 Definitions and abbreviations Throughout this thesis we use a series of terms that have a specific meaning. Next, there is the list of definitions and abbreviations ordered alphabetically. Augmented Reality (AR): It is a specific type of MR where most of the information is real and virtual objects are coherently located onto the real scene. See Section 2.1. Edutainment: Refers to the combination of education and entertainment. It aims to engage students in the learning process throughout playful activities. Handheld AR: Refers to the use of handheld devices to interact with AR applications. Handheld device: It is typically an electronic device that is portable and is intended to be held only with one or two hands. It can be used for m-learning, entertainment, video games, serious games and many other areas. Two of the most common handheld devices are mobile phones and Tablet PCs, that nowadays usually have tactile screens. Head-Mounted Display (HMD): Device that allows the rendering of computer generated imagery in a display close to the eyes. It can allow the visualization of AR from the user’s perspective. Human-Computer Interaction (HCI): It is a very active research area where the interaction with computers is studied, and usually involves highly multidisciplinary studies. It also refers to any kind of possible interaction and communication between a machine and a person. M-learning: Mobile learning. It involves a particular educational model where information is ubiquitous and uses powerful handheld devices to provide rich multimedia experiences. Mixed Reality (MR): It refers to the synthesis of virtual and real imagery that creates a combined scene of virtual and real information in any kind of proportion. See Section 2.1. Mixed Reality Learning Environment (MRLE): Refers to that use MR or AR under an educational framework. Tablet PC: Handheld device that has a relatively large screen around 10” and nowadays typically has a great computational power and the screen is tactile. Tactile User Interface (TacUI): It is an interface where the interaction is performed using a tactile surface, which usually is also a screen so that the user can push directly on the object being displayed. This term was defined by the authors of this thesis and is not usually found in the literature. 1 Chapter 0 Tangible User Interface (TUI): This interface involves an interaction that is performed through the manipulation of physical objects. Virtual Reality Learning Environment (VRLE): Refers to that use a fully VR system under an educational framework. 2 Chapter 1 Introduction 1.1 Motivation Teachers are continuously trying to understand better the process of education to enhance students’ comprehension, and there is a very active body of research in finding better teaching methods using tools that can reach their students at multiple levels (Veenema and Gardner,1996;Tan et al., 2008). The learning experience is a key point in education, and it has been shown that it can become more meaningful when more senses are involved (Sandor and Klinker,2005). From the children’s perspective, a very exciting way of using their senses is through play, which is an important activity to improve and develop children physically, mentally, socially, and emotionally. Play can be used effectively to make use of all of children’s senses to solve problems and to understand their environment in a natural way (Rapeepisarn et al.,2006). Play can help children experience a more meaningful form of learning than other traditional approaches (Gee,2003) and stimulates them to understand new concepts that would otherwise could be found too difficult to grasp (Blecic et al., 2002;Squire et al.,2004). Playful activities serve as a medium to develop capabilities and abilities through active involvement in an amusing way, and enjoyment is important when endeavoring to achieve learning goals, since enjoyability takes an important role in creating mindful learning (Blecic et al.,2002). Educational games integrate learning in a playful environment through participative techniques that help children to develop their knowledge and abilities. They also increase motivation towards study and can also be used to reinforce knowledge acquired in the classroom. Psychologists and philosophers have studied the influence of playing games on the learning process concluding that entertainment is an important factor that helps improving learning (Albert and Mori,2001). Edutainment is a term where education and entertainment converge, and it relies heavily on technology like video games (Pan,2006;Rapeepisarn et al.,2006). As some studies have shown children can benefit significantly from digital educational games, since they can improve their knowledge and skills, and are also stimulated motivated (Fisch,2005;Rigas and Ayad,2010;Shelton and Hedley,2002). M-learning is a research area that involves a new educational model (Fotouhi-Ghazvini et al., 2011). Even though there is no a consensus on the definition of m-learning, it is a term that usually refers to the use of mobile devices in an educational context (Sharples et al.,2002). The new educational model of m-learning has some advantages over other learning methods. In m-learning, the devices used are small, portable and wireless. They make the educational process flexible and adaptable for students, and they are usually cheaper than other devices like desktop computers 3 Chapter 2 from the interviews done, some pretests and posttests about the system developed might have helped the authors to obtain more feedback. In 2010, Chang, Lee, Wang, and Chen presented RoboStage, a mixed-reality learning environment with robots to help students learn new words (Chang et al.,2010). Thirty-six eighth-grade students participated in the study. Four groups were formed. Two of them completed the learning activities using an English textbook and the other two used RoboStage to complete them. The comparison between the two methods showed that RoboStage significantly improved the sense of authenticity of the task and also positively affected learning motivation and performance. The participants felt like they were putting language into real use when using virtual robots. Despite this, no significant differences in terms of learning new words between using the virtual and the mixed-reality environments were found. 2.3 MRLE Education is a field of research that can benefit extraordinarily from technology. The VRLE is well established as an educational tool (Lee et al.,2010), while AR is not as spread yet. AR could be a strong complement to this traditional approach to education, and learners could enrich their experience with a MRLE (Pan et al.,2006) in the classroom. Many VRLE applications have been developed. In 2010, Yang, Chen and Jeng developed a video-capture VR system in a classroom environment for English learning (Yang et al.,2010). The system was tested on 60 students, divided in an experimental group and a control group that experienced traditional english learning. The experimental group perceived visual and auditory feedback, and they could interact physically with the virtual environment instead of the traditional avatar. The authors measured the differences in learning and found that both systems achieved a similar level of learning in the immediate posttest, but interestingly another delayed test showed a significantly higher english knowledge in the experimental group. Moreover, the motivation was also measured, and the results showed that the children rated the game higher in almost all items. The authors concluded that the system effectively enhanced student’s learning motivation and assisted in English long-term learning. In 2009, Lee, Wong and Fung developed a VRLE application to study the effectiveness on students, emphasizing the positive academic effects (Lee et al.,2009). The subject to be studied was frog dissection, and they used the VR program V-Frog. A total of 431 students between 15 ant 17 years old participated in the study who were divided in the experimental group and the traditional classes group using Power Point slides with the biology teacher. The authors found a significant difference in the academic performance, perceived learning and satisfaction. Students showed a better attitude towards using the VR application. The authors advise that VRLE should not be the panacea, but they affirm that VRLE should become part of the everyday life. We can compare the previous VRLEs examples chosen previously to the next AR systems used in classes. In 2011, Connolly, Stansfield, and Hainey presented ARGuing, an Alternate Reality Game designed for a PC to increase the motivation of students in the learning of foreign languages (Connolly et al.,2011). Forty-five participants between 12 and 15 years old took part in the trials. Students played the game in the classroom or at home for 10 days. The study, which aimed at increasing the motivations of secondary school students in the learning of modern foreign languages, showed positive results regarding attitudes, motivation, and perceived learning with evidence suggesting that the system managed to deliver the motivational experience expected by the students. The participants complained about the amount of time involved in completing the tasks and the difficulty of some of the tasks. This might lead to a decrease of motivation. 10 State of the art In 2008, Freitas and Campos presented SMART, which consisted of a TV-show style learning game that was composed of several racquets with 3D augmented reality markers, a web camera, a PC, and displays such as LCD or projectors (Freitas and Campos,2008). Fifty-four students between 7 and 8 years old participated in the trials. The study compared a class using traditional methods and students who used the SMART system. The questionnaires focused on the knowledge questions and did not ask the participants about the usability or the engagement and fun, etc. of the system used. Analyses showed that SMART had better results than the traditional method in weak and average students, but the effect of the experimental system on good students was less noticeable than in the traditional class group. These results should be further examined, since Freitas and Campos did not examined whether or not there were statistical differences between the different groups and gender. While some researches in the field of education state that VR is “extremely close to reality” (Inoue,2007, p. 1), such is not still the case of AR. We attribute this to the fact that it is very modern in comparison to VR, and to the few easy to use, low cost, updated AR tools for educators that can be found. Handheld devices offer excellent capabilities to be used for education, and have a great perspective of future (Billinghurst and Henrysson,2006). In our opinion, Tablet PCs can be an excellent tool that helps AR to have good acceptance as a complement to VRLE. MRLEs should be a natural step towards computer-aided education in the class. Tablet PCs are usually equipped with several sensors and mechanisms for rich Human-Computer Interface. They generally include camera, tactile screen and inertial measure units such as accelerometer and gyroscope. Thus, they can be used in a wide range of education areas, reducing the cost of custom hardware. 2.4 Tactile and tangible interfaces in AR There exists a large body of research TUIs in AR applied to education education. In 2011, Sayed et al. presented ARSC, a low cost visualization tool of 3D objects that could be used online and offline for different subjects reducing the learning time (Sayed et al.,2011). The authors used desktop computers with web cameras to visualize objects on markers that students could move with their hands as a TUI. They tested the system with students between 10 and 17 years old, and 89% of them were satisfied with it. The ARSC set decreased the expenses and increased the visualization ability of the students. In 2002, Sharlin et al. presented cognitive cubes, a TUI system that allowed the manipulation of cubes for the assessment of building 3D figures (Sharlin et al.,2002). This system did not work with AR. The users had to physically connect the cubes to form a shape that was sent through a wire to the computer. The system was tested on adults and compared to paper-and-pencil 3D spatial assessment, and it was concluded that for the cubes improved flexibility, reliability, sensitivity of cognitive ability. In what could be considered as an evolution of the cognitive cubes metaphor, Juan et al. (2010) combined AR with tangible cubes, in which each side there was a marker and no physical connections were needed (Juan et al.,2010). The users wore a Head-Mounted Display (HMD) to visualize the AR scenes, freely manipulating the cubes with both hands. The evaluations compared the AR that displayed videos to traditional cubes with images on the sides, on the subject of endangered animals. The authors tested the system with children from 7 to 12 years old, and they concluded that despite the HMD being uncomfortable, children enjoyed more the AR system. This thesis brings interesting conclusions that could be applied to the continuation of the research on cognitive cubes and other similar metaphors that use tangible markers with video see-through visualization. In 2008, Kim and Maher studied the use of TUIs on spatial cognition, comparing it to a traditional graphical user interface with keyboard and mouse (Kim and Maher,2008). The experimental study 11 Chapter 2 was based on the organization of an interior design office and tested with architecture students. The tangible setup consisted of markers placed on a table representing parts of the office and a vertical monitor with an AR system, while the traditional system was a CAD program. The authors found that TUIs changed users’ spatial cognition and affected the design process. In our research, we have paid special attention to previous work in TacUIs and handheld devices. In 2006, Wagner, Schmalstieg, and Billinghurst presented the collaborative handheld AR game Virtuoso for learning history of art (Wagner et al.,2006). In the game, players had to sort a collection of artworks by date of creation in three different ways: using a paper, a PC and a PDA. The authors did not find significative differences in educational outcomes, but users preferred paper and PDA over the PC to have more working space to collaborate, and they also preferred the AR PDA game over the paper game to have different points of view . In 2007, Schmalstieg and Wagner (Schmalstieg and Wagner,2007) presented Studierstube as a complete handheld AR framework with a case study where students from 12 to 15 years old used handheld devices to explore historical artifacts in the environment of a museum. The students had to select the items on their screens once they had found them, and some pieces of information and multimedia was displayed. The results were very satisfying, and they were very motivated and wanted to extend the game to other exhibitions. 12 Chapter 3 Study 1: Mixed Reality Learning Environments In this chapter, we present a novel study that emphasizes the use of AR as a natural complement for the VRLE model, towards a general acceptance of MRLE in the classroom. Handheld devices help this scheme serving as general purpose computers available for use by other applications. AR has not been explored deeply enough to have full acceptance of use in the classroom. We present an application in which a Tablet PC was used to evaluate our game, working with multimodal interaction provided by a tactile screen and an accelerometer. It can be played in two modes: combining AR and non-AR (NAR), and full NAR. Seventy three children of primary school tested the system. For the learning outcomes, there were no statistically significant differences between both modes, but the AR mode enhanced highly user satisfaction and engagement. This confirms our hypothesis that AR can be an excellent complement to VRLE for the use in the classroom. 3.1 Introduction As far as we are concerned, there is still too much separation between the fields of AR research and education inside the classroom. We believe that the latter could benefit from the great contributions of handheld AR applications. This study has the objective of highlighting this gap and serving to future researchers as a reference to deepen in this direction. Our initial hypothesis is that AR can be a good complement to VRLE for educational purposes to improve learning. With this study, we want to prove or reject this hypothesis. In this research, we propose the case of use of a game. Many computer games have been developed for learning purposes, but very few perform a deep analysis, as several researchers have highlighted (Connolly et al.,2011;Freitas and Campos,2008;O’Neil et al.,2005). Some researchers have also pointed out the lack a coherent theory of learning and underlying body of research in the development of educational applications (Shaffer et al.,2004). In this study, we present a handheld game that not only uses AR, but also combines it with non-AR (NAR) parts –including video games– as a case of VRLE. Video games is a subject widely studied previously. For desktop computers, different subjects can be learnt such as volcanoes (Woods et al.,2004), dinosaurs (Bimber et al.,2001), the relation between the earth and the sun (Shelton and Hedley,2002), mathematics and geometry (Kaufmann,2004), how to play billiards (Larsen 13 Chapter 3 Figure 3.1: Child playing one of the AR games, catching the main character water drop. et al.,2005), organic chemistry (Fjeld et al.,2007), or endangered animals (Juan et al.,2011a). For handheld devices, several educational AR applications have also been presented. For example, for learning heritage temples (Wang et al.,2009), math and literacy skills (O’Shea et al.,2009), or how to recycle (Juan et al.,2011b). The innovate aspect of this study relies on the research of a scenario with practical usability in the classroom. We emphasize the convergence between AR and VRMLE to form a MRLE, as a very suitable tool for educators that can improve the outcomes of VRLE and support meaningful learning. We also believe that Tablet PCs are very appropriate to put MRLE into practice, as they are affordable, allow AR applications and provide multimodal interaction. 3.2 The game To study MRLEs we decided to design and build a game that incorporates AR and VR in a single game. In this section we explain the design principles and educational background that we incorporated to the game and a description of its functionalities and phases. 3.2.1 Game design The game that we have developed is themed on the water cycle. This subject was chosen given a former study in which professionals in education were consulted to determine the subject preferences for educational computer games and their type for children. From the survey, we found that “Nature” was one of the most preferred subjects. Furthermore, we chose the water cycle theme because this topic is covered in the author’s country primary education law. Some works have pointed out the importance of considering national curricula to develop educational computer games (De Freitas and Oliver,2006;Lai-Chong Law et al.,2008). We have taken into account the national curricula for our game, as stated by the national primary education law in the Royal Decree 2211/2007, on July the 14 Study 1: Mixed Reality Learning Environments 12th. In the first section Contents of second cycle of primary education, it establishes: “The water cycle. Explanation of the water existence in different states and how it can change from one state to another through heating or cooling” (BOE,2007, p. 31501). And in the 6th section, it establishes: “Pollutants production, contamination and its environmental impact” (BOE,2007, p. 31502). We followed some of the design characteristics that were identified for virtual learning environments by Mueller and Strohmeier (2010), which referred to virtual environments, but can also be applied to AR systems. One of the highest rated characteristics was “Interaction”, so we took into account several design suggestions from different authors to improve the learnersystem-communication. Liarokapis and Newman, suggested to combine multiple interaction forms (Liarokapis and Newman,2007). Therefore we used accelerometers, a tactile screen, and tangible interaction. The accelerometers and the tactile screen were used to move characters in the screen or to pick objects during the game in order to complete some tasks. There was tangible interaction because markers could be rotated and translated with the hands. Other design principles and suggestions were also followed in order to enhance interaction (Koh et al.,2010), and it has also been recommended mixing several input and output channels (Sandor and Klinker,2005). In our game had camera, accelerometers, and a tactile screen as input channels, and videos, sounds, and graphics as output channels. Henrysson and Billinghurst considered tracking real objects using a camera to have a 6 Degrees of Freedom input (Henrysson and Billinghurst, 2007). We used the camera of the device for tracking markers, and children could select the virtual objects that appeared on them. We also tried to keep interaction techniques as user friendly as possible (Zhou et al.,2008) since it is an important factor to take into account in order to provide an engaging gaming experience (Koh et al.,2010). We also tried to achieve a high degree of naturalness in interaction (Aliakseyeu et al.,2002) by using two-handed interaction instead of one-handed interaction, since children use two hands when playing the mini-games. These mini-games had visual feedback since every action of the children had a reaction in the game. We only used wireless mobile devices, so children were not annoyed by any other instrument. The characteristic of “Learning-process supportive” was also highly rated in Mueller and Strohmeier’s study (Mueller and Strohmeier,2010). As some studies have pointed out, education cannot be improved with only having technology (Fisch,2005;Veenema and Gardner,1996), but technology can include a variety of media that helps understanding and learning concepts (Veenema and Gardner, 1996). In our game, for example, children used the camera in AR mini-games whose perspective was similar to a first-person perspective, making the children embedded agents, becoming part of the game (Dickey,2005). The educational context in which the design of our game is supported consists of two learning theories, which are described next. Gardner’s Multiple Intelligences Gardner’s theory of Multiple Intelligences (Gardner,1983) has become a catalyst and a framework for many current educational strategies. According to Gardner, intelligence is not a unitary element, but it includes different and specific ways of learning and processing information. Kolb described eight types of intelligence through which individuals approach problems and develop solutions. According to the Multiple Intelligences theory, a person has at least eight forms of intelligence (linguistic, logical-mathematical, musical, spatial, bodily-kinesthetic, naturalistic, interpersonal, and intrapersonal), and a correct use of technology can help their development (Gardner,2000). Next we detail the activities proposed in our game for seven of these forms of intelligences, describing the theoretical meaning of the intelligence and how it has been incorporated to our game. 15 Chapter 3 Linguistic. This is the competence to use words in an effective way, both oral and written expressions. It assumes having skill in the use of the syntax, phonetics, semantics and pragmatic uses of the language. In our game, children have to hear and, in some cases, read the directions that the guide character (a drop of water) gives during the game, which allow the children to gradually understand the dynamics of the activity. It is a linguistic activity, like reading a book, listening to a story... The children have to use their language skills to get through the game. Logical-mathematical. This is to have skill in solving logical-mathematical problems. In our game, seven different problems arise that the children have to solve using logical thinking in order to progress in the game. The proposal to form water molecules in their scientific formulation stands out from the rest: H2O. The children have to identify the formula for water and take the atoms in order (two hydrogen atoms and one oxygen atom). Visual-spatial. It includes the ability to perceive and represent the visual-spatial world accurately and to form and manipulate mental images. We propose solving spatial problems in our game through observation and perceptual stimulation of objects from different angles. To do this, four Augmented Reality games were used that allow playing with the physical space and virtual objects using 3D in concepts related to the water cycle. While the children play, they can visualize objects from different angles. Augmented Reality allows virtual objects in to be inserted in the real space so that children can play with the augmented space created. Naturalistic. It consists of understanding the natural world. One of the learning objectives in our game is to bring the water cycle game to children in the second cycle of primary education in a fun and attractive way. The learning contents are designed to help the children understand the water cycle. Intrapersonal. This form of intelligence develops individual and personal knowledge, identity construction and self-esteem. In our game, the media and development of the game have been proposed for a single use in order to respect the times for personal learning, individual working knowledge construction and individual self-esteem. Musical. It is composed of different skills: perception, performance and production. Music perception allows different meaning within a musical composition to be discerned. In our game, music and sound effects have been introduced to allow the children to get into the gameplay and to reinforce feelings of accomplishment and self-esteem related to the upcoming game stages. Bodily-kinesthetic. It is linked to the ability to control our body. The game is conceived as an exploration of space, where the children have to move in order to locate objects (augmented reality games). Kolb’s Experiential Learning Theory For the design of the game, the experiential learning theory of Kolb was used, which states that “learning is the process whereby knowledge is created through the transformation of experience” (Kolb,1984, p. 38). As shown in Figure 3.2, the experiential learning consists of four phases: a 16 Study 1: Mixed Reality Learning Environments Concrete Experience Active Experience Abstract Conceptualization Reflective Observation Figure 3.2: The wheel of Kolb’s learning. concrete experience (do), a reflective observation (observe), an abstract conceptualization (think), and an active experience (plan or testing in new situations). In Kolb’s theory, the wheel that forms the four phases can be started from any point, which can lead to different learning processes. Our game was designed to initiate the learning wheel through a concrete experience. In the reflecting phase, the students think about what the have experienced (what have been seen, heard, or manipulated) and integrate this with their prior knowledge about the water cycle. These previous phases allow abstraction and generalization, adding meaning to the experience. The acquired concepts can be used actively in new situations, and the children can use what they learned in their daily life. As an example of the application of this theory to our game, the player assumes the mission of completing the cycle of water and she has to perform different activities. The player has the concrete experience of collecting suns to raise the temperature, with a reflecting observation with the feedback of the game, and creating an abstract conceptualization as a result of receiving all the information, in this case, evaporation. And what is very important, the student has an active experience using the game. 3.2.2 Description of the game The aim of the game was to reinforce the learning of children in the subject of water, including its composition, the water cycle and water pollution. These topics were shown in the same way they had been studied at school. They saw the major processes of the water cycle. They started with the evaporation phase, when the sun heated up water and turned it into vapor or steam. Then, this water vapor or steam lifted into the air. Next, they saw the condensation phase, which was produced when water vapor in the air got cold and changed back into liquid, forming clouds. The third water cycle phase they saw was precipitation, which occurred when the small drops that formed the clouds got cold. The last water cycle phase was collection, which occurred when water fell back to earth as precipitation. Apart from the water cycle, the children also learned the water composition and about water pollutants. Using AR, the children explored a room looking for the objects requested by the guide character, which in the game was a drop of water. To search for the objects, the children focused the device’s camera on the different markers distributed around the room. Ten different AR markers were used and placed in the activity room, which was decorated with wall posters and images throughout 17 Chapter 3 AR Non-AR Main character Composition Evaporation Precipitation Condensation CoolPollutants Figure 3.3: Flowchart of the AR and non-AR mini-games. the room for a more immersive experience. When the children selected an object, a message was displayed telling them whether the object picked up was the right one. Figure 3.5 shows a child playing the game. Non-AR mini-games (which in our application were virtual reality games) appeared in combination with some AR mini-games. The non-AR mini-games did not use AR, but they used tactile or accelerometer capabilities. The non-AR mini-games usually consisted of children having to collect a certain number of objects to get to the next screen. Between each mini-game (AR and non-AR), video and audio explanations were displayed, describing the rules and goals to complete the next mini-game. They also served as a reward for the children when they had completed a mini-game, showing what they had achieved. This way, we could link all the mini-games together in a continuous story thread. There were seven mini-games in total, which are shown graphically in the flowchart in Figure 3.3. The game started with a video, which introduced the children to it. Next, they searched the room looking for the guide character, a drop of water. This activity served as a tutorial and was the first contact with our AR system. The children learned to focus and to select objects in the AR mini-games. After the guide character had been found, the first task required the children to form water drops from oxygen and hydrogen atoms. The atoms were represented as characters (different colored drops of water) with the letter ‘H’ or ‘O’ on their backs. The children first had to select two hydrogen atoms and then an oxygen atom. Then, a drop of water was formed. Once the children made three drops of water, they went to the first non-AR mini-game. Here, they had to collect twenty suns to evaporate the drops of water they had made before. This mini-game corresponds to the evaporation phase in the water cycle. The suns fell from the top of the screen and the children had to move a water drop that was at the bottom of the screen from left to right using the tactile screen capabilities of the devices. Next, in the second non-AR mini-game, the children had to place ten clouds over a mountain peak tilting the device to use the accelerometer capabilities. The third AR mini-game, which corresponded to the condensation phase of the water cycle, asked the children to find thermometers with low temperatures, which were blue in color. There were also thermometers with high temperatures, which were red in color. This way, they could cool the clouds and produce precipitation. Then, in the collection phase, the children had to collect the drops of water that were falling from the mountain peak (Figure 3.4). In order to complete the task, the children had to collect twenty drops of water by touching them. Finally, the water cycle was completed, but the guide 18 Study 1: Mixed Reality Learning Environments Figure 3.4: Child playing the mini-game of the rain. character discovered a problem. A river was filled with objects that could be pollutants, so another request was made to the children. They needed to find and pick up the pollutant objects using the AR capabilities of the devices. All the mini-games contained a head-up display that showed the current status. 3.3 Materials and methods For the evaluation of our study, we needed to build both custom hardware and software to provide a multimodal interface. This section explains the hardware built and the software developed. 3.3.1 Hardware For the evaluations we used the handheld device HP Slate 500 Tablet PC. The dimensions of the device were 23x15x1.5 cm, and it had a weight of 0.68 kg. It used an Intel Atom Processor Z540 working at 1.86 GHz, and a 2 GB DDR2 SDRAM memory cell. For the graphics, it operated with a built-in GPU Intel Graphics Media Accelerator 500. Among its interfaces we could find a 8.9” capacitive touch-screen and an outward facing 3 Megapixel camera capturing at 30 fps. The HP Slate 500 had a pre-installed 32 bit version of Windows 7 Professional. There were severe performance issues with the graphic controllers used for the integrated GPU. In our early tests running the basic OpenSceneGraph application osgviewer with a model consisting in about 8000 polygons, the frame rate was about 5 frames per second (fps). After some web research, we found that some low power devices using Windows suffer from this problem. It is due to the inadequate adaptation of the default drivers to the hardware, which in our case was the combination of Itel GMA 19 Chapter 3 # AR 1st NAR 1st t p 1 0.97 ±0.17 1.00 ±0.00 1.04 0.30 2 0.97 ±0.17 0.95 ±0.22 0.51 0.61 3 0.94 ±0.23 0.97 ±0.16 0.66 0.51 4 0.97 ±0.17 0.95 ±0.22 0.51 0.61 5 1.00 ±0.00 0.97 ±0.16 0.96 0.34 6 0.97 ±0.17 0.89 ±0.31 1.29 0.20 Table 3.4: Means and standard deviations of questions of the AR and NAR games played in the first place, and t-test analysis. d.f.=71. # AR 1st NAR 1st t p 7 4.86 ±0.35 4.84 ±0.36 0.18 0.86 8 4.66 ±0.47 4.66 ±0.53 0.01 0.99 12 4.09 ±1.00 4.34 ±0.98 1.09 0.28 16 4.43 ±0.65 4.39 ±0.81 0.19 0.85 17 4.77 ±0.42 4.68 ±0.52 0.78 0.44 18 4.89 ±0.40 4.92 ±0.27 0.44 0.66 19 4.76 ±0.60 4.75 ±0.39 0.17 0.86 Table 3.5: Means and standard deviations of questions of amusement and satisfaction of the AR and NAR games played in the first place, and t-test analysis. d.f.=71. play again, and the 7.9% were not sure. 3.5.2 Satisfaction Outcomes In order to determine if the experiment influenced participants in regard to the level of amusement experienced and satisfaction between the two modes of game, the variable satisfaction was created. It condensed all information in the concerning questions, giving each one of them the same weight. An unpaired t -test showed that the satisfaction ratings in QAR1 (mean 4.73±0.47 ) were not statistically different from QNAR1 (mean 4.66 ±0.63 ) ( t[71] = 0.28,p=0.78 ). The detailed results of each question are shown in Table 3.5, where several paired t -tests were performed. There were no significant differences in any isolated question; the scores were very high for both games, from what we can say they enjoyed the experience in both cases and were engaged to learn in a similar manner. In order to determine the effect of the order in which both game modes was played, several tests were conducted. Table 3.6 shows the unpaired t -tests for AR games played the first and the second time. These shared questions measure the general satisfaction with the game –questions 7, 16 and 19–, and also the specific thoughts about AR –questions 9, 10 and 14–. AR had excellent acceptance, and children enjoyed it very much. We can see the high values of question 9, asking if they liked this technology, and question 10, asking if they liked the games that used AR. From the notes of our staff that was with the children during the evaluations, we obtained very good impressions, specially in the AR part, which was more encouraging to be a dynamic activity. Table 3.7 shows the unpaired t -tests for NAR games played the first and the second time with 26 Study 1: Mixed Reality Learning Environments # AR 1st AR 2nd t p 7 4.86 ±0.35 4.82 ±0.39 0.47 0.64 9 4.86 ±0.35 4.82 ±0.39 0.47 0.64 10 4.77 ±0.48 4.89 ±0.31 1.29 0.20 14 3.86 ±0.72 4.00 ±1.00 0.68 0.50 16 4.43 ±0.65 4.50 ±0.94 0.37 0.71 19 4.76 ±0.30 4.78 ±0.32 0.25 0.80 Table 3.6: Means and standard deviations of questions of the AR game played first and NAR in second place, and t-test analysis. d.f.=71. # NAR 1st NAR 2nd t p 7 4.84 ±0.36 4.86 ±0.35 0.18 0.86 16 4.39 ±0.81 4.51 ±0.55 0.72 0.47 19 9.50 ±0.78 9.41 ±0.83 0.45 0.66 Table 3.7: Means and standard deviations of questions of the NAR game played first and NAR in second place, and t-test analysis. d.f.=71. questions about satisfaction. The results showed no significant differences between both game modes. From this data, we can deduce that the order of playing did not significantly affect the scores of the games. The preferences for AR or NAR were measured in question 20 (Figure 3.8). After playing in AR mode first and then NAR, 65.7% of the children chose AR over NAR, while 94.7% of them chose the same when playing AR in second place. The augment in the proportions was statistically significant ( χ2[1,73] = 9.90,p=0.002 ), but the difference of proportions between AR and NAR in the worst case (AR first) was also significant ( χ2[1,73] = 6.91,p=0.008 ), which means that AR was preferred even if we do not take into account the effects of the order of the games. Relating this to results from question 11, we can see that a trend in which AR was preferred over other alternatives, and this is emphasized when it is the last technology to be used. Among the reasons given in question 21 to prefer AR over NAR, we could find: AR NAR NAR 2nd AR 2nd Preference (%) 020 60 100 Figure 3.8: Preferences after the last game played (question 20). 27 Chapter 3 AR Accelerometer Tactile AR 1st AR 2nd Preference (%) 020 40 60 Figure 3.9: Preferences after the AR game played the first and the second time (question 11). Accelerometer Tactile NAR 1st AR 1st Preference (%) 020 40 60 80 Figure 3.10: Preferences after the first game played (question 15). • “I can see things of the game in the real world.” • “Because you learn a lot.” • “Because it was more entertaining and fun.” • “You had to move and you were not seated all the time.” • “Because it was easier.” The reasons given in question 21 to prefer NAR can be summarized in: • “Because I like to play seated.” • “Because it was more comfortable.” 3.5.3 Interaction The preferences of the different types of interaction were measured in question 11, considering the accelerometer games, tactile games and AR games using the device in a video see-through configuration. The question was asked after playing games in AR mode (results in Figure 3.9). When AR was played in first place, the percentages of preferred technology was: AR (54.3%), accelerometer (34.3%) and tactile (11.4%), but when AR was played the last, the order was: AR (73.7%), accelerometer (21.1%) and tactile (5.3%). AR was always the favorite, and it raised when it was played in last place. The proportions of AR in both cases were significantly different at level α=0.1 ( χ2[1,73] = 2.99,p=0.08 ). We believe this difference is due to the big impact AR caused to the children. As a consequence, accelerometer and tactile proportions dropped similarly without significant differences. Question 15 asked about how easy it was to use the tactile screen and the accelerometer after each game the first time they were played (Figure 3.10). On one hand, the accelerometer was the most appreciated, and was higher NAR (71.1%) than in AR (57.1%). On the other hand, the tactile screen raised from NAR (28.9%) to AR (42.8%). In the NAR mode the proportions between both interaction methods were statistically significant at level α=0.01 ( χ2[1,73] = 13.47,p<0.001 ), but in AR mode, no statistical differences were found ( χ2[1,73] = 1.43,p=0.23 ) and the differences may be due to chance. Therefore we can deduce that the use of AR stimulated the use of tactile interaction to the point that it was not significantly different from the accelerometer interaction. 28 Study 1: Mixed Reality Learning Environments 3.6 Conclusions In this work we have presented a study that emphasizes the use of AR as a natural complement for VRLE, which is MRLE. Educational AR researches typically focuses in Human-Computer Interfaces and usability, and the applications are frequently intended for museums. To our knowledge, AR has not yet been seriously studied as a complement to VRLE to be used in the classroom for a long-term use. We believe that MRLEs are very suitable for the classroom and we encourage future educational researches to take it into consideration. Handheld devices, and Tablet PCs in particular are an excellent tool for MRLEs. They commonly provide all the sensors needed to build interactive applications and multimodal interfaces, such as camera, tactile screen and inertial measure units. This makes them to be an exceptional tool that can be used for a wide spectrum of interactive educational applications with more possibilities and versatility than standard desktop applications. Moreover, Tablet PCs are usually comparable in prize to desktop computers, both being low cost solutions. We developed a MRLE application for primary school children, which consisted of a game about water that included multiple interaction forms (tactile screen and accelerometer). It could be played in a combined mode with AR and NAR mini-games, or in full NAR mode. After playing to the game, children’s knowledge was statistically higher than in the pretest, but no significant differences were found between AR and NAR modes. However, the AR mode enhanced user satisfaction and engagement highly. This confirms our initial hypothesis that AR is an excellent complement for VRLE and that it can improve some of its outcomes. In our game, this combination of AR and NAR games throughout the story thread and the links between them like the main character was very appreciated. Playing in AR mode caused very good impression to the children, who improved motivation and were encouraged to be more dynamic during the activity, not being so perceivable in the NAR mode. We could see the big impact that AR caused, as the underlying interaction –AR in a handheld device as a video see-through– was significantly more preferred than accelerometer and tactile screen, specially when it was played in second place. There was a significant preference for the accelerometer interaction over the tactile interaction in NAR mode. However, the introduction of AR encouraged tactile interaction, and the difference was no longer significant. During the evaluations we observed that a percentage of the children tended to touch the capacitive screen with their fingernails, causing an apparent malfunction that would not have happened with a resistive screen. Although capacitive screens have better acceptance in the general public than resistive screens, the latter should not be trivially discarded when developing applications for education purposes with primary school children. With regard to future work, we believe that more research in this direction is needed. MRLE in the field of education is still in an early stage. It could be used for many subjects, including Natural Science, Mathematics, History, Technology and outdoor activities. Furthermore, more engaging games and serious applications that use different input channels (AR marker tracking, tactile screen, accelerometer, etc.) could be developed with current handheld devices. 29 Chapter 4 Study 2: Impact of handheld devices in AR for children In this chapter, we present an AR game for a smartphone (iPhone) and a Tablet PC, designed to reinforce children’s knowledge about the water cycle. The game included different interaction forms like a tactile screen and accelerometers, and during the gameplay AR mini-games were combined with non-AR mini-games for better immersion. We present a study from a HCI perspective to determine the differences in satisfaction and interaction caused by the use of the two handheld devices. Seventy-nine children from 8 to 10 years old participated in the study. The two devices were enjoyed and the children were very motivated. However, despite that our initial hypothesis was that the Tablet PC would be more appreciated, from the results we observed that the different characteristics (screen size and weight) of the devices did not influence significantly children’s engagement and satisfaction. We analyze the reasons of these results and the conclusions from this experiment. 4.1 Introduction In this study, we present a mobile AR game for learning about the water cycle, water composition, and water pollution. The content of the game is the same as in the first study, described in Section 3.2, but contrary to that study where we compared two versions of the game, we are now only interested in using the original version that contains a mixture of AR and VR mini-games. In this case, the game was compared using two handheld devices with different characteristics: a smartphone (iPhone) and the Tablet PC used in the first study, described in Section 3.3.1. The game combines AR and non-AR mini-games. Using AR, the children explored a room looking for objects by focusing the device’s camera on different markers. The non-AR mini-games usually consisted of children having to collect a certain number of objects using the sensors in the device to get to the next screen. Our game combined AR mini-games with non-AR mini-games for better immersion. Furthermore, the game combined different forms of interaction, including the tactile screen and the accelerometer in the multimodal interface, as it has been suggested by some authors (Liarokapis and Newman,2007). The main objective of this study is to observe if one device had more influence than the other on the participants regarding the acquired knowledge, satisfaction, and interaction. The primary hypothesis was that there would be significant differences between playing in an iPhone or a Tablet 31 Chapter 4 iPhone game Q1 Tablet PC game iPhone game Tablet PC game Q2 Q2 Q3 Q3 Group A Group B Figure 4.1: Scheme of the games and questionnaires where each children followed one path. PC device regarding acquired knowledge, satisfaction, and interaction, and that children would prefer the Tablet PC mainly due to its larger screen size. Based on the developments and studies mentioned in the state of the art, in Section 2.2, to our knowledge our work is the first one that compares two different mobile devices with different characteristics (screen size and weight) to see its impact on children’s knowledge reinforcement, satisfaction and interaction. 4.2 Design of the evaluations 4.2.1 Participants Seventy-nine children from 8 to 10 years old –with a mean age of 8.70 ± 0.70– took part in the study: 42 boys (53%) and 37 girls (47%). These children were attending a summer school They were attending the Escola d’Estiu (Summer School) at the Polytechnical University of Valencia.. 4.2.2 Measurements Three questionnaires were used for the validation. The first one was the pretest, and the other two were filled out after playing the first game and the second game as shown in Figure 4.1. Table 4.1 shows the relation of questions for each questionnaire. The questions are very similar than the used in the first study, and we refer the reader to Section 3.4.2, for a detailed explanation. The differences in this study are the distribution of questionnaires. The pretest (Q1) was composed of six questions designed to evaluate how much the children knew or remembered about water from school (composition (Questions #1, #2), cycle (Questions #3, #4, #6), and pollutants (Question #5)). The second questionnaire (Q2) was composed of 19 questions. The first six questions were the same ones the participants answered in Q1 to be able to compare them. The rest of the questions were about participant satisfaction and interaction with the game. The third questionnaire (Q3) was composed of eight questions. Some of the questions were the same as in Q2, which allowed us to compare the two devices. In question #6, the children had to select which device they preferred to play the game: What device did you like the best? a) iPhone; b) Tablet PC. In question #7, the participants explained why they preferred one device over the other. In the last question, children described what they liked the most of the whole experience. 4.2.3 Procedure The children who participated in this study were randomly assigned to one of two groups: 32 Study 2: Impact of handheld devices in AR for children # Q1Q2Q3 Question 1• • Do you remember what comprises water? 2• • Of the components that you are going to read which ones do not belong to water? 3• • Do you remember what helps water to evaporate? 4• • Once water drops are in the clouds, what do they need to go down to the land or to the sea? 5• • Check every object you think is pollutant. 6• • Could you tell us how the water cycle is? 7• • Did you have fun? 8•Did you like that the guide character guided you during the game? 9• • Did you like to see how objects appeared on the black squares? 10 •Did you like playing to games that use what you found on the black squares? 11 •What did you like the most? [AR/Tactile/Accelerometer] 12 •Do you think you have learned new things? 13 •Would you like to play again to learn about new subjects? [Yes/No/Maybe] 14 • • Did you find it easy to find objects on the black squares? 15 • • What did you found the easiest? [Tactile/Accelerometer] 16 •How easy to play did you find the game? 17 •Did you understand the rules of the game? 18 •Would you like to learn new things at school with this system? 19 • • Please, rate the game. 20 •What did you like the most? [AR/NAR] 21 •Why? (referring to #20) Table 4.1: Numbered questions and their appearance in each questionnaire. The first six questions have custom answers. The last two questions are hand written. Answers in brackets are a summary of the possible choices (categorical data). The rest of answers follow a Likert scale. Group A) Children who played the iPhone game first and then the Tablet PC game. Group B) Children who played the Tablet PC game first and then the iPhone game. Both groups had a similar number of subjects: 41 children were assigned to group A, and 38 to group B. Figure 4.1 shows the procedure of both groups graphically. As can be observed, before playing any game, each child filled out the pretest Q1 and some instructions were given to the children about how to play the game. Then, the first group played the iPhone game. After completing the game, they answered the posttest Q2. Then, these children played the Tablet PC game and filled out the ending questionnaire, Q3, when they had finished playing. The second group played the Tablet PC game first, and after completing the game, they also filled out Q2 questionnaire. Then, these children played the iPhone game and also answered the Q3 questionnaire when they had finished playing. In our study, the content evaluation protocol was established in a way similar to the one applied to assess the contents in the classroom. In the classroom of the second cycle of the primary education in the country of the authors of this work, the usual established dynamic is to teach a subject and then evaluate the level of learning of the content. Two mirrored rooms were used for the evaluations. Each room had two identical playing areas where two children (one of each group) could play simultaneously but individually. There was no interaction between them, and there was also a person with each child to guide them and to 33 Chapter 4 Factor d.f. F p Effect size (η2 G) Grade 1 2.14 0.15 0.03 Gender 1 0.02 0.83 <0.01 Device 1 1.05 0.31 0.01 Grade:Gender 1 2.05 0.16 0.03 Other interactions 1 <0.09 >0.36 <0.01 Table 4.2: Multifactorial ANOVA for the variable of satisfaction. N = 79. clarify the possible doubts during the whole activity. The questionnaires were filled out in the same room where the children played. The children were encouraged to answer all questions without any pressure to avoid the influences of answering the pretest on the results of the posttest. If they did not know the answer it was considered to be completely normal, the children were not informed whether their answers on the pretest were correct. Thus, the children did not acquire any knowledge by answering the pretest; they only learned during the game. 4.3 Results The variable satisfaction was created to combine the answers of several questions (the mean of answers to question 7, 8, 9, 16, 17, 18, and 19), giving us a measure of the degree of engagement and enjoyment with the game. The overall rating of this variable was very high (mean 4.77 ±0.23 ), indicating that the children were highly satisfied with the game. The variable was analyzed using a multifactorial ANOVA with the gender, grade and the device factors (Table 4.2). As we can see, there are p-values and small effect sizes, which means that there were no significant differences for any of the factors, including the device used. We could deduce that since the game was very appreciated and enjoyed very similarly by all the groups studied, motivation to learn was increased by using the game. With regard to the introduction of AR in the game (question 10), the overall rate for the enjoyment of AR games was very high (mean 4.85 ±0.39 ), and an ANOVA test showed that all the groups studied appreciated it in a similar manner without significant differences: grade ( F[1,71] = 0.09,p=0.77,η2 G<0.01 ), gender ( F[1,71] = 0.90,p=0.35,η2 G=0.01 ), and device ( F[1,71] = 0.58,p=0.45,η2 G<0.01 ). This result is very positive because it means that the use of AR can be spread over all of the factors studied without any restriction. We also measured how easy it was for the children to use the AR system (question 14). The global rating was high (mean 4.29 ±0.83 ), which means that children found it easy to play AR games, and all the groups experienced it similarly, since no significant differences were found in the grade, gender, group, the order in which the devices were used, or the device used. The results of the mixed design analysis are shown in Table 4.3, where the order and device factors were within subjects and the rest were between subjects. According to the p-values and the effect sizes, none of the factors had significant differences. Similarly to the previous result, this result is very appreciated by the authors because it means that the introduction of AR was very positive for all the children and all the devices homogeneously. A Chi-squared test for question 20 revealed that the preference for the iPhone or the Tablet PC significantly differed between children who finished playing with one of the devices ( χ2[1,79] = 12.08,p<0.01,Cramer’s V=0.42 ). After analyzing the results, we could see that children tended 34 Study 2: Impact of handheld devices in AR for children Factor d.f. F p Effect size (η2 G) Grade 1 2.52 0.12 0.03 Gender 1 0.77 0.38 <0.01 Group 1 2.25 0.14 0.02 Order 1 2.24 0.14 <0.01 Device 1 1.22 0.27 <0.01 Interactions 1 <1.81 >0.15 <0.02 Table 4.3: Mixed design ANOVA for the ease of use of the AR system. N = 79. Group A Group B iPhone Tablet PC Preference (%) 0 20 40 60 80 100 Figure 4.2: Preferences for the favourite device in groups A and B. Tablet PC iPhone Accelerometer Tactile Preference (%) 0 20 40 60 80 100 Figure 4.3: Preferences for the easiest technology after playing each device. to choose the device they had used the last time. This difference is shown graphically in Figure 4.2, where 71.43% of the children in group A chose the Tablet PC, and 70.27% of the children in group B chose the iPhone. Another test revealed that there were not significant differences in this tendency for the devices ( χ2[1,79] = 0.05,p=0.83,Cramer’s V=0.05 ). Therefore, we could conclude that there was a bias towards preferring the last device used, but the effect of that bias was equivalent for the Tablet PC and the iPhone. Thus, we could conclude that there was no significant difference in the preferences for the Tablet PC and the iPhone, and that the differences were due to the order of playing. To interact with the device, the children thought that it was easier to use the accelerometer rather than to touch the tactile screen (question 15), as we can see in Figure 4.3. There were significant differences in the proportions when children played the Tablet PC ( χ2[1,79] = 14.6,p<0.001,h= 0.64 ), but not after playing the iPhone ( χ2[1,79] = 0.911,p=0.34,h=0.176 ). Comparing both devices, we found no significant differences in the proportions of the preferred interaction on the two devices ( χ2[1,79] = 1.69,p=0.19,Cramer’s V=0.12 ). However, we believe that the higher weight of the Tablet PC could have influenced this result decreasing the tactile rates because some children had some difficulties holding the device while touching the screen. Question 11 evaluated the preferences of different types of technologies in the mini-games: AR, tactile screen, and accelerometers (Figure 4.4). There were no significant differences found 35 Chapter 5 Figure 5.2: Set used for the evaluation consisting on a TUI and a TacUI in the handheld device. Hardware for the TUI. We built a home-made plastic rotatory base resembling to a gearwheel with four teeth that could be manipulated to rotate it comfortably. the marker was placed in the centre. Software. We decided to use OpenSceneGraph (OSG) toolkit 2.9.5 to develop the system to take advantage of its high capabilities to import, animate and render 3D objects with high performance in C++ language. The AR registration was achieved with the OSG plugin osgART 2.0 RC 3, that used the ARToolKit library (Kato and Billinghurst,1999), version 2.72.1. osgART provided simple access to the camera, and to certain OSG nodes that applied the corresponding transformation matrices associated to markers when they were recognized. 5.3 Design of the evaluations The developed game described in Section 5.2 was tested by a group of children. This section details the participants, the measurements and the procedure designed for the evaluations. 5.3.1 Participants A total number of 51 children from 8 to 10 years old –with a mean age of 8.65 ±0.74 – took part in the study. The gender distribution was: 29 boys (57%), and 22 girls (43%). (More information about the participants will be provided after the anonymous review). 5.3.2 Measurements Each child tested the two types of interaction (TacUI and TUI) so that they could compare them at the end. This leads to paired samples and repeated measures in the statistical analysis. In each game the time to achieve the task was measured, and the child had to fill in a questionnaire afterwards. Consequently, we used four different questionnaires (QTc1, QTc2, QTn1, and QTn2; Tc stands 42 Study 3: Tactile and tangible interfaces TUI TacUI # QTn1QTn2 QTc1QTc2 Question 1• • • • Did you enjoy playing the game? 2• • Did you like how the Taj Mahal appeared on the black square? 3• • Would you like to play again to search other objects? [Y/N/M] 4• • • • How easy did you find the game to play? 5• • Would you like to use the wheel control in other games? 6• • • • Please, rate the game. 7• • What did you find the easiest? [Tactile/Tangible] 8• • What did you like the most? [Tactile/Tangible] 9• • Why? (referring to #7 and #8) Table 5.1: Numbered questions and their appearance in each questionnaire. The last question is hand written. Answers in brackets are a summary of the possible choices (categorical data). The rest of the answers follow a Likert scale. for tactile and Tn for tangible) during the evaluations, which contained the proper questions to be asked after using each type of interaction in first or second place. However, each child had to fill in only two questionnaires, one for each interaction. We can see the list of all the questions and the organization of the questionnaires in Table 5.1. The questionnaires for the first game, QTc1 and QTn1, are similar. Both have satisfaction measures, and ask about AR and about the interaction method. The questionnaires of the games played in second place, QTc2 and QTn2, include questions to compare the interaction methods. Some of the questions follow a Likert scale presented as: a) Very much; b) Quite a lot; c) Somewhat; d) Few; e) Nothing, with a numerical equivalency linearly ranged from 5-a) to 1-e). Other questions have categorical answers, as they ask to choose among several options (#3, #7 and #8). The last question is hand written and gives the children an opportunity to express their opinion about the game openly. 5.3.3 Procedure The children were divided in two groups depending on the order in which they would play with both interfaces: Group A) Children who used the TacUI first and then the TUI. Group B) Children who used the TUI first and then the TacUI. The children were randomly assigned to the groups and counterbalanced: 26 children were assigned to group A and 25 to group B. During the evaluations, only one child was playing at a time. She played two times to the same game of counting objects, changing the interaction form. The time to complete the task was recorded for posterior analysis. After playing each game, the child answered a questionnaire. We can see a graphic that summarizes the process in Figure 5.3. A member of our staff guided the child during the process and explained her the tasks to do. After the measures, all data was analyzed sing the open source statistical toolkit R1. 1http://www.r-project.org/ 43 Chapter 5 QTc1 QTn1 QTn2 QTc2 Group A Group B Tactile TactileTangible Tangible Figure 5.3: Questionnaires and games that children played in groups A and B. ● ●● ● Tangible 2 Tactile 2 Tangible 1 Tactile 1 0 20 40 60 80 100 120 Time (s) Figure 5.4: Box plot for the time to complete the task in the four combinations of tactile and tangible interfaces played the first and the second time. 5.4 Results The first step to analyze the results was the identification the possible outliers. From a visual inspection at the box plot of the time to complete the task, shown in Figure 5.4, we detected four outlier points. These outliers correspond to three different participants, one of whom scored a very high value with the two interfaces he played. These participants were two boys and one girl, and all of them were 8 years old. To avoid possible misleading conclusions, their times and questionnaires answered were completely excluded from the analysis. 5.4.1 Time analysis The time to complete the task of counting the objects in the scene was measured and studied as an independent variable. Comparing this variable in the first interaction that the two groups used, we found that the time spent when using the TacUI (mean 43.21 ±13.12 ) was statistically significantly lower than when using the TUI (mean 50.30 ±13.16 ), as an unpaired t -test revealed ( t[46] = 1.83,p=0.04,Cohen’s d=0.53 ). In this analysis we follow the guidelines of Cohen to interpret the effect size, which state the approximate meaning of d : 0.2 small effect size, 0.5 medium effect size, and 0.8 large effect size (Cohen,1988, pp. 25–26). Therefore, the interaction factor had 44 Study 3: Tactile and tangible interfaces Factor d.f. F p Effect size (η2 G) Interaction 1 4.94 0.03** 0.04 Order 1 33.13 <0.01*** 0.20 Age 2 0.23 0.79 <0.01 Gender 1 0.49 0.49 <0.01 Group 1 0.09 0.76 <0.01 Table 5.2: Summary of the mixed design ANOVA analysis for the time to finish the task. * p<0.1 ; **p<0.5; ***p<0.01. N = 48. a medium effect on the time the children spent to complete the task when they played to it for the first time. A medium effect size was interpreted by Cohen as a magnitude perceivable by the “naked eye”. Consequently, we can conclude that the TacUI was significantly faster to use than the TUI, not having in consideration any other factor. We also performed a mixed-design ANOVA analysis to take into account the two interactions the same child used, and to consider several factors simultaneously. In a mixed-design ANOVA, some factors are between-subjects (i.e. all of the measures come from different subjects or it is an observed factor like the age or gender) and other factors are within-subjects (i.e. some measures are repeated from the same subject). In our analysis we took into account the factors of age, gender, group (A or B), interaction (TUI or TacUI) and order (1st game or 2nd game). Therefore, the factors of age, gender and group were between-subjects, and the factors of interaction and order were within-subjects. To perform the analysis we did two different mixed-design ANOVAs, one with the interaction factor as within-subject and the age, gender and group factors as between-subjects, and another one with the order factor as within-subject and the age, gender and group factors as between-subjects. This separation in two different tests is due to the strong relation between the interaction factor and the order factor. These two factors contain the same subjects but classify them according to different criteria, and since the mixed-design ANOVA needs that each subject has a measure of all combinations of the levels of the within factors, it is not possible to use the two factors simultaneously, but they can be used independently. Nonetheless, thanks to the design of our variables, this separation has no effects in our analysis because the two within-subjects factors classify the subjects independently from the between-subjects factors. Thus, the two mixed-design ANOVAs obtain the same results for the factors of age, gender and group, and this is the reason why we present the results of the two analysis in the same table that summarizes the two statistical tests (Table 5.2). In this table we can see that there are significant differences in the order and interaction factors. As both factors have two levels only, a simple post-hoc analysis comparing the mean of each level revealed that the users took less time to complete the task using the TacUI than using the TUI with a significant difference. Similarly, we concluded that the users took more time to complete the task the first time they played than the second time. No significant differences were found for the age, gender and group factors. To enhance our analysis with a measure independent from the sample size, we also calculated the effect size used using the generalized eta squared, η2 G (Olejnik and Algina,2003). This effect size measure has been proven to be more suitable for mixed-design analyses than other traditional alternatives such as the eta squared, η2 , or the partial eta squared, η2 P , because it can consider the repeated measures of some subjects along the within-subjects factors. Olejnik and Algina stated that, unlike η2 and η2 P , η2 G is a measure consistent across a wide variety of different statistical designs. 45 Chapter 5 # Factor d.f. F p Effect size (η2 G) 1 Interaction 1 0.19 0.67 <0.01 4 Interaction 1 0.10 0.75 <0.01 6 Interaction 1 2.08 0.16 <0.01 1 Order 1 2.96 0.09* 0.02 4 Order 1 0.00 1.00 <0.01 6 Order 1 1.41 0.24 <0.01 Table 5.3: Summary of the mixed design ANOVA analysis for the common questions. * p<0.1 ; **p<0.5; ***p<0.01. N = 48. Bakeman also stated that it seemed appropriate to use Cohen’s guidelines for η2 for multi-factor analysis: 0.02 small effect size, 0.13 medium effect size and 0.26 large effect size (Bakeman,2005). These guidelines for multi-factor analysis (Cohen,1988, pp. 413–414) should not be confused with the guidelines for one-way analysis (Cohen,1988, pp. 285–288), which have lower values. Unfortunately, it is very infrequent to find effect sizes reported by researchers in the field of HCI. To our knowledge this is the first study in the field to report an analysis using η2 G , and it was not possible for us to compare our study with previous results. Our results in Table 5.2 show a small to medium effect size for the interaction factor, which is consistent with the previous result that analyzed the same factor only for the first game played, revealing that the second game introduced noise in the analysis. Nevertheless, a small effect size “in practice represents the true order of magnitude of the effect being tested” (Cohen,1988, pp. 413–414). There is also a medium to large effect size for the order factor, revealing an important learning from the first task to the second task, which was an expected effect in this experiment. 5.4.2 Satisfaction The answers from the questionnaires were also studied. Table 5.3 shows the results for the interaction and order factors (other factors were also studied but are not shown since no relevant results were found) for questions 1 (enjoyability), 4 (ease of play) and 6 (global game rate). All of these questions have very high scores: #1 4.60 ±0.64 , #4 4.58 ±0.66 and #6 4.64 ±0.49 . Moreover, there are no significant differences in questions 4 and 6 for the studied factors. The only arguable point is in question 1, where there is a relatively high p-value and a small effect size. A post-hoc analysis reveals that the possible difference would mean that the second game was more enjoyed than the first game. We believe that the difference is meaningful and not only caused partially by chance, but also by the order factor, as we can see from the answers to the question 3. In this question, 93% of the children said they would like to play again after the first game, and we could see during the evaluations that they were very excited to play the second game, what in our opinion has affected the results in question 1. The analysis of the interaction factor in question 3 shows that all of the 25 children who played the TacUI first would like to play again, while 20 children would also like to play again after the TUI, and 3 were undecided. However, here are no evidences of significant differences over the interaction factor ( p=0.10 , Fisher’s exact test), which means that both interactions engaged the children highly. The children appreciated the AR very much, as the high scores to the question 2 reveal for 46 Study 3: Tactile and tangible interfaces Tactile Tangible Tactile 2nd Tangible 2nd Preference (%) 0 20 40 60 80 Figure 5.5: Percentages of selection for the easiest interaction (question 7). Tactile Tangible Tactile 2nd Tangible 2nd Preference (%) 0 20 40 60 80 Figure 5.6: Percentages of selection for the preferred interaction (question 8). the tactile game (mean 4.80 ±0.40 ) and for the tangible game (mean 4.91 ±0.28 ). It is very interesting to note that no significant differences were found between both games ( t[46] = 1.10,p= 0.28,Cohen’s d=0.32 ), which is an excellent result because it means that the use of AR was very similarly enjoyed with both interactions. Since the TUI was built ad-hoc, we asked the children if they would like to use the tangible controller in other games (question 5) as a measure of their attraction and evaluation. The scores to this question were very high when the TUI was the first interaction (mean 4.65±0.70 ) or the second one (mean 4.84±0.37 ). No significant differences were found ( t[46] = 1.15,p=0.25,Cohen’s d= 0.33 ), from what we can deduce that the TacUI did not influence the appreciation of the tangible game. 5.4.3 Interaction We also compared the two interactions studied in this work from the children’s point of view after having tested both of them. The ease of use and preferable form of interaction is analyzed next. Question 7 asked the children to choose the type of interaction they thought it was easier to use. As we can see in Figure 5.5, there is the effect that children chose more the last game they used, which is a common tendency that we have experienced in previous studies with children. When the TacUI was the last one to be used there was a 70% of children who preferred it to the TUI, but when the TacUI was the last one to be played, the TUI preferences dropped to 48%. When the children used the TacUI in second place, there were significant differences in the proportions of TacUI and TUI ( χ2[1,48] = 5.57,p=0.02,h=0.80 ) with a large effect size (Cohen,1988, pp. 184–185). This result evinces the preference of the TacUI. The difference is, however, no longer significant when the TUI was used last ( χ2[1,48]<0.01,p=1.00,h=0.08 ). From the difference in the significances we can conclude that the tactile screen caused a more powerful impact on the children. The preference for the two types of interaction was measured in question 8. Figure 5.6 shows graphically the results in percentages. It is apparent that the situation from question 7 is present in this case as well. The most extreme proportions are found in the tactile game played the second time, where the tactile game was chosen by the 78% of the children, and the tangible game by the 22%. In such case there are significant differences and a very large effect size ( χ2[1,48] = 12.52,p<0.01,h=1.20 ). When the tangible game was played last, the differences are not significant, but there is a medium effect size, revealing more apparent differences than in 47 Chapter 5 question 7 (χ2[1,48] = 1.76,p=0.18,h=0.51). The reasons given by the children in question 9 on why to prefer the TacUI can be catalogued in the next: • “Because it is easier” • “Because it is faster to touch the screen” • “Because it is funnier” • “Because it is more comfortable” The reasons given by the children that preferred TUI can be catalogued in the next: • “Because it is slower to turn the wheel” • “Because my fingers slipped in the screen” • “Because it was the first time I used something like that” Given the answers, it seems that children of these ages are not very used to tactile screens, and some of them found it a little difficult to control their actions in the screen. In addition, we saw that some of them tended to touch the screen with the fingernails, which was unfruitful since the screen was capacitive. The TUI offered them a more solid interface that they could grab. Despite this, most of the children preferred the TacUI. 5.5 Conclusions We have presented a study that compares TacUIs and TUIs in handheld AR. Our study was centered in the field of education, but many conclusions can be extrapolated to other areas of HCI. To our knowledge, this is the first study that analyses these two interactions and compares them in this way. We designed an system to compare TacUIs and TUIs under very similar circumstances. The handheld device was used in a video see-through mode to visualize the augmented world, taking some of the advantages of HMDs, but resulting in a much more comfortable way to the children. This metaphor was chosen for being one of the most used in education. Our proposal is very well suited for schools, since handheld devices are affordable and multipurpose, so they can replace desktop PCs and enhance their value contributing to create richer multimedia experiences for the students. In the experiment we evaluated, the TUI consisted on a rotatory base on top of which the AR marker was placed. The children could rotate the marker to see the 3D objects from different perspectives. This interaction was compared to the TacUI, where the rotation was calculated from the horizontal movements of the finger on screen. We firstly measured the differences between the two interactions from an objective perspective, analyzing the differences on the time the users spent to achieve the objective of counting the hidden objects on the scene. The tests indicate that it took significantly less time to the children to use the TacUI than the TUI. As for the children’s perspective, we could see a very high level of satisfaction with the game using the two interfaces, and no significative differences were found. The children were very engaged with the game. All of them wanted to play again after using the TacUI, and most of them after using the TUI. However, there was no evidence that the interface used influenced this engagement. The AR system appealed the children very much without being affected by the interaction method. 48 Study 3: Tactile and tangible interfaces The children seemed to prefer the TacUI mainly for being easier and faster to use, and some of them also thought it was more comfortable. On the contrary, the TUI was preferred in some cases, surprisingly for very similar reasons. These children found the screen too fast to use and too slippery, and they preferred the more stable interface that the TUI was. In general, more children preferred the TacUI, although this difference was not always statistically sustainable. Despite preferring the TacUI, the appreciation of the TUI was very high, so we can discard that the preference of TacUI is due only to disliking the alternative. Finally, it is very possible that in older ages we would find a high increase on preference of the TacUI, as the children’s physical skills improve and do not have troubles with the screen. In our statistical analysis we have used the effect size η2 G , which provides a better understanding of the meaning of the significative differences that may be obtained analyzing the p-values in ANOVA analyses. We encourage researchers to use this measure, that could bring more uniformity to the studies, making them more easily comparable. With regard to future work, we think we have now the basis to open the study to more sophisticated interactions. We find suggestive to investigate the possibilities in terms of interaction of using handheld devices more dynamically in activities that require more movement and holding the devices with the hands. 49 Chapter 6 Concluding remarks 6.1 Conclusions In this thesis we have carried out a research about the implications of handheld devices and AR in an educational context and the consequences in HCI. Through three studies we have fulfilled the principal and the secondary objectives. A playful activity was developed, themed in natural sciences, concretely in the water cycle, composition and pollution. The design was carried out following the theories of Gardner and Kolb as the educational background with great success. The motivation of children was very high, and they learned much while playing. We programmed the game and evaluated it in handheld devices, making use of accelerometers, the tactile screen, and the camera, thus providing a multimodal interface that appealed to the children. The game had AR mini-games and non-AR mini-games as specified, and the consequences of using a MRLE or a VRLE were studied concluding that the introduction of AR was an important step for current VRLEs used in the classroom. Also, two different handheld devices were tested and compared when children played the game with them, and we saw that both of them were similarly effective. Furthermore, a study that compared tactile and tangible interaction was presented, providing data to support that it could be preferable to use tactile interfaces in some situations. All the studies presented a decent number of subjects and several measures were taken, such as learning outcomes, satisfaction and interaction. Finally, we presented a statistical analysis of the data captured and provided a critical discussion. In the first study we compared a MR version of the game and a VR version, in a learning environment. The children who tested the two systems learned much, but in a similar amount. This evinces that technology itself is not enough to improve education. We saw our hypothesis accomplished when we saw the great results in motivation. The main conclusion we can get is that AR can improve VRLE, since it enhances the learning experience, but it does not imply a radical change in education. We also tested the game in the second study comparing two handheld devices with different sizes and weights. Despite our initial hypothesis, the Tablet PC would be more engaging than the smartphone, and the interaction would be easier, we found that in our experience, both devices were similarly appreciated. We perceived in children an extraordinary ability to adapt to any kind of environment. In the third study, we compared Tactile User Interfaces (TacUIs) with a Tablet PC and Tangible User Interfaces (TUIs) with an interface built ad hoc. The interface represented a very common 51 Chapter 6 Kerawalla, L., Luckin, R., Seljeflot, S., and Woolard, A. (2006). Making it real: exploring the potential of augmented reality for teaching primary school science. Virtual Reality, 10(3):163–174. Ketamo, H. (2003). An adaptive geometry game for handheld devices. Educational technology & society, 6(1):83–95. Kim, M. J. and Maher, M. L. (2008). The impact of tangible user interfaces on spatial cognition during collaborative design. Design Studies, 29(3):222–253. Koh, R., Duh, H., and Gu, J. (2010). An integrated design flow in user interface and interaction for enhancing mobile ar gaming experiences. In IEEE International Symposium On Mixed and Augmented Reality-Arts, Media, and Humanities (ISMAR-AMH), pages 47–52. IEEE. Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development. Englewood Cliffs, NJ: Prentice-Hall. Lai-Chong Law, E., Kickmeier, D., and Holzinger, A. (2008). Challenges in the development and evaluation of immersive digital educational games. Lecture Notes in Computer Science, 5298:19–30. Larsen, L., Jensen, R., Jensen, K., and Larsen, S. (2005). Development of an automatic pool trainer. In Proceedings of the 2005 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology, ACE’05, pages 83–87. ACM. Lee, E. A.-L., Wong, K., and Fung, C. (2009). Learning effectiveness in a desktop virtual realitybased learning environment. In Proceedings of the 17th International Conference on Computers in Education, Asia-Pacific Society for Computers in Education, Hong Kong. Lee, E. A.-L., Wong, K., and Fung, C. (2010). How does desktop virtual reality enhance learning outcomes? a structural equation modeling approach. Computers & Education, 55(4):1424–1442. Liarokapis, F. and Newman, R. (2007). Design experiences of multimodal mixed reality interfaces. In Proceedings of the 25th annual ACM international conference on Design of communication, pages 34–41. ACM. Mahamad, S., Ibrahim, M. N., and Taib, S. M. (2010). M-learning: A new paradigm of learning mathematics in malaysia. International journal of computer science & information Technology (IJCSIT), 2(4):76–86. Mcconatha, D., Praul, M., Chester, W., and Lynch, M. J. (2008). Mobile learning in higher education: an empirical assessment of a new educational tool. Educational Technology, 7(3):15–21. Milgram, P. and Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems, E77-D(12):1321–1329. Mueller, D. and Strohmeier, S. (2010). Design characteristics of virtual learning environments: an expert study. International Journal of Training and Development, 14(3):209–222. Olejnik, S. and Algina, J. (2003). Generalized eta and omega squared statistics: measures of effect size for some common research designs. Psychological Methods, 8(4):434–447. O’Neil, H., Wainess, R., and Baker, E. (2005). Classification of learning outcomes: Evidence from the computer games literature. The Curriculum Journal, 16(4):455. 58 BIBLIOGRAPHY O’Shea, P., Mitchell, R., Johnston, C., and Dede, C. (2009). Lessons learned about designing augmented realities. International Journal of Gaming and Computer-Mediated Simulations, 1(1):1–15. Pan, Z. (2006). Editorial: Special issue on edutainment (E-learning and game). Computers & Graphics, 30(1):1–2. Pan, Z., Cheok, A., Yang, H., Zhu, J., and Shi, J. (2006). Virtual reality and mixed reality for virtual learning environments. Computers & Graphics, 30(1):20–28. Prensky, M. (2003). Digital game-based learning. Comput. Entertain., 1(1):21–21. Rapeepisarn, K., Wong, K. W., Fung, C. C., and Depickere, A. (2006). Similarities and differences between “learn through play” and “edutainment”. In Proceedings of the 3rd Australasian conference on Interactive entertainment, IE ’06, pages 28–32, Murdoch University, Australia, Australia. Murdoch University. Rigas, D. and Ayad, K. (2010). Using edutainment in e-learning application: an empirical study. International Journal of Computers, 1(4):36–43. Sandor, C. and Klinker, G. (2005). A rapid prototyping software infrastructure for user interfaces in ubiquitous augmented reality. Personal and Ubiquitous Computing, 9(3):169–185. Sayed, N. A. E., Zayed, H. H., and Sharawy, M. I. (2011). ARSC: Augmented reality student card. Computers & Education, 56(4):1045 – 1061. Schmalstieg, D. and Wagner, D. (2007). Experiences with handheld augmented reality. In 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, ISMAR ’07, pages 3–18. Schwabe, G. and Göth, C. (2005). Mobile learning with a mobile game: design and motivational effects. Journal of Computer Assisted Learning, 21(3):204–216. Shaffer, D. W., Squire, K. T., Halverson, R., Gee, J. P., et al. (2004). Video games and the future of learning. Phi delta kappan, 87(2):104. Sharlin, E., Itoh, Y., Watson, B., Kitamura, Y., Sutphen, S., and Liu, L. (2002). Cognitive cubes: a tangible user interface for cognitive assessment. In Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, CHI ’02, pages 347–354, New York, NY, USA. ACM. Sharples, M., Corlett, D., and Westmancott, O. (2002). The design and implementation of a mobile learning resource. Personal Ubiquitous Comput., 6(3):220–234. Shelton, B. and Hedley, N. (2002). Using augmented reality for teaching earth-sun relationships to undergraduate geography students. In 1st IEEE International Augmented Reality Toolkit Workshop, page 8. IEEE. Shelton, B. and Hedley, N. (2004). Exploring a cognitive basis for learning spatial relationships with augmented reality. Technology, Instruction, Cognition and Learning, 1(4):323–375. Squire, K., Barnett, M., Grant, J. M., and Higginbotham, T. (2004). Electromagnetism supercharged!: learning physics with digital simulation games. In Proceedings of the 6th international conference on Learning sciences, ICLS ’04, pages 513–520. International Society of the Learning Sciences. 59 Chapter 6 Tan, K. T. W., Lewis, E. M., Avis, N. J., and Withers, P. J. (2008). Using augmented reality to promote an understanding of materials science to school children. In ACM SIGGRAPH ASIA 2008 educators programme, SIGGRAPH Asia ’08, pages 1–8, New York, NY, USA. ACM. Thornton, P. and Houser, C. (2005). Using mobile phones in english education in japan. Journal of Computer Assisted Learning, 21(3):217–228. Ullmer, B. and Ishii, H. (2000). Emerging frameworks for tangible user interfaces. IBM Systems Journal, 39(3.4):915 –931. Uzunboylu, H., Cavus, N., and Ercag, E. (2009). Using mobile learning to increase environmental awareness. Computers & Education, 52(2):381–389. Van Krevelen, D. and Poelman, R. (2010). A survey of augmented reality technologies, applications and limitations. International Journal of Virtual Reality, 9(2):1–20. Veenema, S. and Gardner, H. (1996). Multimedia and multiple intelligences. American Prospect, 7(29):69–76. Wagner, D., Schmalstieg, D., and Billinghurst, M. (2006). Handheld AR for Collaborative Edutainment. Advances in Artificial Reality and TeleExistence, pages 85–96. Wang, K.-H., Chen, L.-C., Chu, P.-Y., and Cheng, Y.-M. (2009). A study on the design of augmented reality user interfaces for mobile learning systems in heritage temples. In Proceedings of the 3rd International Conference on Virtual and Mixed Reality: Held as Part of HCI International 2009, VMR ’09, pages 282–290, Berlin, Heidelberg. Springer-Verlag. Woods, E., Billinghurst, M., Looser, J., Aldridge, G., Brown, D., Garrie, B., and Nelles, C. (2004). Augmenting the science centre and museum experience. In Proceedings of the 2nd international conference on Computer graphics and interactive techniques in Australasia and South East Asia, GRAPHITE ’04, pages 230–236, New York, NY, USA. ACM. Yang, J. C., Chen, C. H., and Jeng, M. C. (2010). Integrating video-capture virtual reality technology into a physically interactive learning environment for english learning. Computers & Education, 55(3):1346–1356. Zhou, F., Duh, H., and Billinghurst, M. (2008). Trends in augmented reality tracking, interaction and display: A review of ten years of ismar. In 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, ISMAR ’08, pages 193–202, Washington, DC, USA. IEEE Computer Society. 60