Utilization of Augmented and Virtual Reality in Geoscience

GeoScience Engineering Vol. 68 (2022), No. 1 geoscience.cz pp. 22–32, ISSN 1802-5420 DOI 10.35180/gse-2022-0066 UTILIZATION OF AUGMENTED AND VIRTUAL REALITY IN GEOSCIENCE Věroslav HOLUŠA , Filip BENEŠ , Michal VANĚK VSB – Technical University of Ostrava, Faculty of Mining and Geology, Department of Economics and Control Systems E-mail: veroslav.ho[email protected] ABSTRACT In today's age of modern information technologies (IT), it is no wonder that modern visualization tools and techniques are finding their way into areas such as geological and environmental sciences. The market offers a wide range of software resources for use and especially for the creation of scenes in virtual reality. The aim of our article is to evaluate a number of tools for creating virtual reality (VR) content and evaluate their practical applicability. In the experiments, several tools for creating virtual reality scenes were tested and these tools were compared in terms of their parameters. The conclusion of the article shows the results, while Photomodeler and RealityCapture were the most suitable software for the academic sphere. On the contrary, Bentley ContextCapture software seems the least suitable. Keywords: Augmented reality; Benchmarking; Geoscience; Virtual reality. 1 INTRODUCTION Virtual reality is increasingly intertwined with the world of science and academic sphere. It is used for the purposes of teaching and creating teaching aids. Vergara et al. [1] described in their study the spread of virtual reality learning environments in schools and described the processes by which these didactic tools can improve the effectiveness of the educational process. In his work, Fernandez [2] presents and discusses the role of virtual and augmented reality in education. Lee et al. [3] explore the possibilities of using the Google Cardboard VR in classrooms as a platform to present learning material. Specifically, Lee's team examined how VR perceptions differed from perceptions of the traditional flat screen format. Wang [4] examined the impact of integrating augmented reality into the curriculum of a particular university course. Salah et al. [5] focus on the teaching and training for industry needs, especially in view of the emerging trends of Industry 4.0. Visualization technologies, especially virtual reality, are emphasized here in order to introduce a sustainable way of training and educating young students. Buyuksalih and his team [6] created a three-dimensional space (3D) virtual model of the Incegiz caves, where they used the method of scanning using a 3D scanner. They edited the images from the 3D scanner in the RiScan Pro software, where they obtained a photo-realistic visualization. Together with the 3D scanner, they used a Canon EOS 7D Mark II digital camera with a resolution of 20.2 Mpx. Thanks to this configuration, they provided a panorama resolution of 422.5 Mpx at the entrance to the caves. Autodesk ReCap Pro software was used to create the 3D model. The generated models were then imported into the Unity3D development environment and tested on the HTC Vive Pro VR glasses. Sang and Xue [7] developed an application for virtual reality education, where they focused on the natural environment. They used a flying drone with a digital camera and a GPS sensor to capture the area. The collected data were then stored in a spatial database, which they used to create a 3D model using photogrammetry. They presented the result of their work to students who can acquire mental perception of the field environment, knowledge of the field, mastery of skills in field geoscience and expand the scope of knowledge. 23 GeoScience Engineering Vol. 68 (2022), No. 1 geoscience.cz pp. 22–32, ISSN 1802-5420 DOI 10.35180/gse-2022-0066 There are a large number of scientific teams that incorporate virtual reality in the preparation of teaching aids in academia. Therefore, we see a strong need to explore tools suitable for creating virtual scenes suitable for academic environments. The market offers a wide range of software resources for use and especially for the creation of scenes in virtual reality. The main goal of our article is to evaluate a number of tools for creating VR content and evaluate their practical applicability. The means for creating the content of virtual scenes are parametrically similar on many levels, such as hardware requirements, user complexity, user base, price, number of export/import types, user interface, manufacturer support, availability in Czech language, etc. At the beginning of the work, the authors asked themselves the following research question: “Is it possible to select the optimal tool for the academic environment based on the comparison of parameters? Which one is the most suitable for academic environment?” 2 POTENTIAL OF VIRTUAL AND AUGMENTED REALITY In order to select the appropriate parameters for assessing software tools, we first decided to conduct a search for meaningful applications of augmented reality (AR) and virtual reality in the field geoscience-related industries. Most of these fields have a similar one in the composition of individual departments at the Faculty of Mining and Geology of VSB – Technical University of Ostrava. 2.1 Mining rescuers training in VR The use of virtual reality in the training of mine rescuers is beginning to be used in Australian mining [8]. Innovations are reflected in the training of mine rescuers. There is a strong effort in this sector to apply preventive measures to reduce mine accidents. The use of information technology, especially virtual reality for the training of mine rescuers and other workers in the mining industry, is given a lot of space here. Figure 1. VR simulator CAVE system [8] The main advantage of using virtual reality is the simulation of extraordinary events. These can be, for example, mine fires, gas explosions, coal dust explosions in a mine, mine earthquakes, gas bursts and the rescue of workers [8]. 24 GeoScience Engineering Vol. 68 (2022), No. 1 geoscience.cz pp. 22–32, ISSN 1802-5420 DOI 10.35180/gse-2022-0066 2.2 Augmented reality in the field of maintenance and of mining machines AR can also be used for machine maintenance and repair. An essential part of the mining site is the maintenance of various tools. With AR, a worker can have the data of any machine or device in digital form. This data can be used to analyze the current situation of a particular machine and decide on its maintenance. This will increase the life of the equipment. Live video chats with remote experts can be arranged for consultation on working on a specific task. An expert would use a camera to see what an AR user is seeing. Using virtual 3D holograms, which the expert can control, s/he would discuss with the user specific areas and components of the machine being repaired [9]. The use of AR remote assistance ensures the maintenance and repair of machines in the most efficient way possible. In the event of a breakdown of any machine, all the information necessary for its repair is served directly to the site technicians. Through detailed information and enriched with pictures or pdf files, site technicians can follow step by step the process of how to fix a certain part. All this is done in a simple and fast way since the use of smart glasses enables hand-free performance. Figure 2. Augmented Reality in the Mining Industry maintenance [9] 2.3 3D scanning to obtain a virtual model of the mine Today drones are used not only for entertainment, but also in many areas of the commercial sphere. Drones can scan the conditions of buildings, fertilize fields or create maps in mines. If we use these devices, we get a 3D model of the mapped area. Figure 3. Obtaining a 3D model of a mine using a drone [10] Combinations of depth sensors, lidar and photogrammetry techniques are used to obtain the shape of the terrain. Drones for terrain scanning have their application especially in surface mines. Recently, there have been several cases of using a mapping drone in the corridors of a deep mine. The use of this technology can be expected mainly in non-gassing mines. In gaseous mines with methane emissions, the drone would have to meet stricter requirements for ATEX protection and its production and operation would be several times more expensive. 25 GeoScience Engineering Vol. 68 (2022), No. 1 geoscience.cz pp. 22–32, ISSN 1802-5420 DOI 10.35180/gse-2022-0066 2.4 Visualization of a surface mine For example, if we already have a drone-scanned terrain down. We can visualize it using augmented or virtual reality and be able to set spatial connections to scale much better than on a two-dimensional space (2D) monitor or map drawing. It is literally possible to visualize these plans even in the middle of the living room with the help of glasses for augmented reality or watching the space via a mobile phone. Figure 4. Visualization of a surface mine using Hololens [11] If we want to elevate the management meeting over the operation and performance of the mine, we can also use tools for modeling and visualizing the operation of the mine in augmented reality. The entire management can then be introduced to performance tables, plans for expanding mining, machine performance, employee movements and a lot of other information that is important for decision-making. Thanks to the fact that in hololens glasses, all participants can share one model and at the same time see here what an individual participant is looking at, it is clear which part of the visualized model he is talking about. Such a model allows zooming and navigation using fly-through and voice or gesture control. 2.5 Visual support for mining Maptek mining company [12] is among the first companies using visualization via Augmented Reality during mining operations, such as support in real-time decision-making process in the opencast mine. Visual guide assistance through AR makes it possible to significantly increase the accuracy and efficiency of drilling. When applying AR to aid drilling practices, it allows the transmission of necessary information by remote experts. Drilling operators also have a wider field of information by improving the accuracy of their work during this difficult process. Increasing accuracy during this process enables the reduction of potential damages and the avoidance of various risks. Figure 5. Maptek solution with augmented reality [12] 26 GeoScience Engineering Vol. 68 (2022), No. 1 geoscience.cz pp. 22–32, ISSN 1802-5420 DOI 10.35180/gse-2022-0066 2.6 Educational virtual tour of a coal mine If a coal mine ends its activities and is closed, it can still serve as an excellent teaching aid for future generations. Thanks to photogrammetry and other tools for scanning spaces, we are able to capture the entire map of the area, including textures, and transfer them to the environment of scenes for virtual reality. Users can then enter this scene at any time from the comfort of their homes and explore underground spaces and view how mining has taken place. Students may be accompanied by a virtual educator giving them a professional explanation, or this explanation may be pre-recorded in certain places on the virtual scene and played when the student visits it. Figure 6. Virtual scene of a closed mine Users will begin their subterranean adventure at the surface of the coal preparation plant before making their way past the coal mixing dome and lamp storage, and entering the underground railroad system. 2.7 Visualization of measured values invisible by human eye In the mine, which is equipped with a number of sensors of physical quantities and concentrations of gases that cannot be seen by the human eye, it is possible to display these values graphically using augmented reality. Some values of these quantities can be modeled using a computational fluid dynamics (CFD) program and their result then blended with the real world using a smartphone or mobile phone [13]. Figure 7. Demonstration of AR model in the case of degassing chimney; contours of mole fraction of CH4 [13] Efficient visualization tools should respect cognitive and perceptual properties of the human brain. Visualization aims to improve the clarity and aesthetic appeal of the displayed information and allows a person to understand large amounts of data and interact with it in more natural way [13]. 27 GeoScience Engineering Vol. 68 (2022), No. 1 geoscience.cz pp. 22–32, ISSN 1802-5420 DOI 10.35180/gse-2022-0066 3 METHODS AND MATERIAL The mutual comparison of specialized software for the creation of 3D models of objects, which are obtained in the real world by the method of photogrammetry, and the determination of the most suitable software for the academic sphere was performed using multi-criteria evaluation. Specifically, an approach based on determining the benefit (value) of individual variants was applied in the evaluation. The benefit of the assessed variants is determined as a weighted sum of partial evaluations of variants with respect to individual criteria, which expresses the following relationship (1). (1) where: Hj – overall evaluation of the j-th variant, vi – weight of the i-th criterion, hij – partial evaluation of the j-th variant with respect to the i-th criterion, n – number of evaluation criteria, m – number of variants. [16] The basis for evaluating variants is criteria matrix, see Figure 8, which characterizes the individual variants using selected criteria. In our case, these 12 criteria were chosen: K1 – HW requirements, K2 – User complexity, K3 – Price per license, K4 – Multilicense availability, K5 – Availability in Czechia, K6 – Number of export types, K7 – Number of import types, K8 – User interface, K9 – Manufactorer support, K10 – User base, K11 – Trial/Demo availability, K12 – Opensource. The authors decided to build weights using the pairwise comparison method, the so-called Fuller's method. The weight of individual criteria is determined on the basis of the so-called Fuller's triangle, within which all criteria are compared. The weights of the criteria are then created by dividing the number of preferences of the individual criteria and the total sum of preferences. In order for the preferences in pairwise comparison to be less affected by the subjective perception of individual criteria, the authors approached 10 experts who deal with the given issue. Individual preferences in pairwise comparison are the result of brainstorming. For the criterion of HW requirements, a typical PC set for the academic sphere was selected, current as of 09/2021 according to internal DNS regulations. Specifically, these are the following technical specifications: CPU Mark at least 13000 points, for example AMD Ryzen5 2600, 16GB RAM, 512GB SSD + 1TB HDD, external graphics card 4GB. There are many software tools for creating 3D models suitable for use in VR. The set of tested software was created after consultation with experts in VR and 3D modeling, and the actual testing of tools on a real project. So the following software tools came out of the whole range. 3.1 3DF Zephyr 3DF Zephyr is a commercial photogrammetric and 3D modeling software. It is developed by the Italian 3DFLOW team, which first launched it in 2014 and has since been continuously updated until the current version 6.009, which is current as of October 2021. It is a complete photogrammetric SW package, which contains a number of tools for the most accurate processing of 3D models. It enables 3D reconstruction not only from photos, but also from video by automatically sorting out individual usable images. 3DF Zephyr only works under Windows Vista and later. 28 GeoScience Engineering Vol. 68 (2022), No. 1 geoscience.cz pp. 22–32, ISSN 1802-5420 DOI 10.35180/gse-2022-0066 3.2 Reality Capture It is a software tool that creates 3D models from unordered photos. Its most common areas of current use are heritage sites, full body scanning, the gaming industry and VR creation. The main advantage of this SW is the automatic calibration of photographs, assignment of unambiguous coordinates in space, very friendly and intuitive environment. You can also use the command line and other software development tools. It is also possible to mix photo images with laser scans. Created 3D models can be exported to major formats such as OBJ, XYZ, DXF, DAE, etc. [14] 3.3 Autodesk ReCap Pro It is a tool for intelligent conversion from photos and scans to 3D data using a point cloud. It is possible to photograph or scan artifacts, buildings, reliefs, even the whole city. Data is stored in a point cloud (millions of points) and Autodesk ReCap Pro processes that amount of data, including visualization measurements and preparation for other Autodesk programs such as AutoCAD, Revit, or Autodesk Inventor. Everything takes place in the following four phases. [15] Scanning, where data from a laser 3D scanner is stored in a point cloud. Import, where Autodesk ReCap Pro converts scanned files to Reality Capture Scan (RCS) format for other Autodesk programs. During this indexing, the raw data is converted to Autodesk ReCap Pro (the number of points can be set at the beginning). The third step is organization, cleanup, and analysis, where various methods are available to organize the indexed data and delete or hide it in the point cloud. The scene can then be viewed in various color modes and lighting. Finally, there is the connection phase, where indexed data from scans and projects can be opened or connected to other programs to create a realistic 3D model. ASC, CL3, CLR, E57, FLS, FWS, ISPROJ, LAS, PCG, PTG, PTS, PTX, RDS, TXT, XYB, XYZ, ZFS, and ZFPRJ files can be indexed. Autodesk ReCap Pro then creates RCS (one unit of measurement file), RCP (project file), and PCG, PTS, and E57, OBJ export formats. 3.4 Meshroom Meshroom is free open source software that has an intuitive environment and is easy to use. It includes a large number of settings, thanks to which a very good result of the reconstructed building can be achieved. Photographs are processed using the photogrammetric framework for computer vision AliceVision. This framework is the result of collaboration between several universities, laboratories and Mikros Images. The Czech Technical University in Prague is one of the universities involved in the development. Generating photogrammetric models is easy and intuitive here. Literally, it is just a matter of tagging the source photos, dragging them into the software window and then clicking to start the process. Of course, there are great options for setting up the model creation process, while it is possible to achieve a higher quality of the resulting model compared to automatic setting. Meshroom is available for Windows, OSX and Linux operating systems. It requires CUDA library support – so it is suitable for machines with nVidia graphics cards. The software allows you to choose between three modes of model creation. Standard reconstruction is suitable in cases where we have a set of photos, without the possibility of taking and adding more pictures. Extended reconstruction is appropriate in a situation where we have completely taken all photos, but if necessary, it is possible to add more photos. This option is especially useful in cases where we want to improve an already finished model. The inserted photos are automatically synchronized with the already used photos, so that, for example, a blurred or missing space of the model can be filled. Live Reconstruction mode is controlled by photos inserted in the source folder at the time of shooting. Thus, a live reconstruction gradually creates a 3D model of a scanned object or space. The use of this option is ideal if we do not know how many photos are needed to create the desired quality of the created model, or if we require the best possible result and we can adjust the settings of the scanning device. 29 GeoScience Engineering Vol. 68 (2022), No. 1 geoscience.cz pp. 22–32, ISSN 1802-5420 DOI 10.35180/gse-2022-0066 3.5 BentleyContextCapture It is a proprietary software for creating 3D models using the method of photogrammetry. It also provides the ability to use the software as a service, so the user does not need to have their own powerful modeling device. It is more user-intensive software for more advanced users. It contains a large number of adjustable options to achieve the best possible final model. The software is suitable for larger teams and projects that can contain resolutions up to 300GPx. However, the purchase price corresponds to this, which makes it a tool only for professional use, as there is no free version. 3.6 AgisoftMetashape The software is a solution for photogrammetric processing of digital images. Output 3D data can be used for work in geographic information system (GIS) applications, for documentation of cultural heritage, or for creating visual effects. It is an automated data processing system that is suitable for both novices in the field of photogrammetry and for professionals, which allows them to set parameters for specific tasks and different types of data. 3.7 Photomodeler It is a software application that performs modeling based on photographs and near-range stereo photogrammetry – it creates 3D models and measurements from photographs. This software is used for near and aerial photogrammetry. Close-range photogrammetry (CRP) can mean photographs are taken from the ground with a hand-held camera or taken from a UAV / drone at a relatively low altitude. Photo modeler and CRP are used to perform measurements and modeling in agriculture, archeology, architecture, biology, engineering, manufacturing, film production, forensics, mining, inventory volumes. 3.8 Regard3D It is an open source application that is freely available under the MIT license. It was developed as a hobby project by a Swiss programmer who decided to make it available to the public free of charge. The program draws on open source libraries, which it combines in various ways to achieve the best possible result. Application development was suspended in 2019 and has not been updated since. 4 RESULTS The starting point for performing a multi-criteria evaluation is the criteria matrix, see Figure 8. Figure 8. Criteria Matrix 30 GeoScience Engineering Vol. 68 (2022), No. 1 geoscience.cz pp. 22–32, ISSN 1802-5420 DOI 10.35180/gse-2022-0066 As we can see in Figure 8, the created set of criteria consists of quantitative and qualitative criteria. In addition, there are criteria for different units. In the case of quantitative evaluation, the evaluation was performed using a scale in which 5 points indicate that the level of fulfillment of the given criterion is fully ensured and, conversely, 1 point indicates that the level of fulfillment does not meet the given criterion. In order to be able to express the utility of individual variants on the basis of input data, it is necessary to transform the criterion matrix into the so-called simple utility matrix. (Figure 9), which replaces absolute values with relative values. We perform this transformation by determining the most advantageous variant for each criterion and assigning the value 100. Subsequently, the other values of the individual criteria will be recalculated. The resulting sum is then a partial evaluation of the assessed variants, because the individual criteria have the same weight. Figure 9. Simple Utility Matrix The weights of the evaluation criteria were determined in accordance with the methodology using the Fuller triangle. Figure 10. Determination of Weights