Rapid Deployment of Visual Path Following via Average Representation

PREPRINT Rapid Deployment of Visual Path Following via Average Representation Zdenˇ ek Rozsypálek∗1 ·Tomáš Rouˇ cek1 ·Jiˇ rí Ulrich · 1Tomáš Krajník1 Preprint Abstract The advances in deep learning for image processing greatly impacted mobile robots’ ability to navigate using vision-based techniques. There are many applications in which the robot is required to follow a specific visual cue. One newly emerging method for autonomous navigation along a consistent visual cue is Visual Teach and Generalise (VTAG). It is a unified approach to extract suitable features for path following in repetitive and structured environments. However, a relatively large number of data samples is required for the time-consuming training of a detector tied to a specific visual cue. We introduce an improved version of VTAG that does not need any neural network training and significantly reduces the requirements for the number of data samples. The core idea is to use an average representation of images collected during short supervised traversals, capturing characteristics of repetitive paths. The average representation is created in a latent space crafted by contrastive learning with the linear matching scheme and can be computed in real time, even on a CPU. The capability of rapid deployment, precision, and robustness of the presented method is evaluated in multiple field experiments performed in different environments. Keywords Path Following ·Visual Navigation ·Machine Learning in Robotics 1 Introduction Mobile robotics has emerged as a hot research topic, driven by the increasing demand for autonomous systems operating in various environments. The advent of advanced sensing technologies, coupled with improvements in computing 1Czech Technical University in Prague, FEE, Czech Republic ∗E-mail: [email protected] Pad Shared weights CNN CNN Embedding Horizontal cross-correlation Histogram (1, 65) Embedding Pad (T, 3, 384, 512) (1, 3, 384, 512) (1, 16, 8, 64) (1, 16, 8, 128) AVG (T   1) Training images Live image Heading control Average map representation Fig. 1: Our method creates an average representation from T training images. This representation is then cross-correlated with a live camera feed. The resulting histogram is used to control the robot’s heading during navigation. power, has enabled robots to understand their surroundings, make decisions, and perform tasks autonomously. Many mobile robots are equipped with cameras, as they are light, cheap, and low-power. However, long-term appearance changes and repetitive environments pose major challenges to vision-based mapping and localisation methods [1]. These challenges led to the development of visionbased navigation methods that do not require building of consistent metric maps for navigation. Visual Teach and Repeat (VT&R) [2] methods strictly divide mapping and navigation into two phases. An operator manually guides the robot in the teaching phase along a desired trajectory. In the repeat phase, the robot follows 2 Zdenˇ ek Rozsypálek∗1et al. the recorded trajectory using previously captured visual features. Visual servoing-based variants of VT&R simplify the navigation stack by utilising image sequences instead of metric maps, eradicating errors associated with traditional mapping. These approaches reduce the requirements on the robustness of the feature matching [3] while keeping the navigation error bounded [4,5]. The handling of environmental variations is critical for long-term, reliable navigation [6], and VT&R is an attractive framework thanks to its robustness. VT&R methods are reliable and easy to deploy, but manually guiding the robot through every possible path is timeconsuming, e.g., [7] mentions that mapping a 0.08 km2urban park took four days. To speed up the mapping, some researchers augmented map-based VT&R with vision-based path following [8]. However, the time reduction in mapping was offset by the time required to design and tune detectors for repetitive paths. Many environments where the deployment of mobile robots would be highly beneficial, such as warehouses, automated farms, or hotels, are composed of elements of similar appearance. The framework that takes advantage of the repetitiveness of visual elements within the VT&R paradigm is called Visual Teach and Generalise (VTAG) [9]. Instead of mapping the entire environment, the VTAG framework creates a detector of an arbitrary visual cue (representing a typical path) consistent throughout the training set. This enables the robot to follow previously untraversed paths, effectively removing the need to map the entire operational area while preserving the simplicity and robustness of VT&R [5]. The deployment scenarios suitable for VTAG are further specified in Section 3.2. The existing VTAG method can reduce the mapping time, but still requires a large number of data samples used for time-consuming neural network finetuning on a specialised computational server. In this paper, we present a method that can create a detector of repetitive visual cue on the fly, making it available instantaneously after the teaching phase. Instead of the fine-tuned neural network detecting the visual cue, we use an aggregated robust representation for the path traversal. The method not only eliminates the neural network finetuning, but also reduces the number of required training images by two orders of magnitude. A diagram describing the presented method is shown in Figure 1. The simplification of the deployment process compared to the current version of VTAG is visualised in Figure 4. To summarise, our main contribution is that the presented method offers similar functionality as the original approach [9] while: a) Removing the requirement for fine-tuning the neural network to follow a visual cue. b) Significantly reducing the number of map images required for a successful traversal. c) Speeding up the deployment process by creating a representation of a visual cue in real-time. d) Demonstrating capabilities of VTAG in multiple scenarios with extended autonomy. The contribution (d) mainly extends the trustworthiness of the generalisation capability of the method, (a), (b), and (c) improve usability by reducing the deployment time as the neural network does not need to be fine-tuned any more. Additionally, without fine-tuning, we can avoid unpredictable changes of the latent space and enhance the reproducibility. In this paper, we first discuss relevant research and existing methods. Then, we define the symbolism and terminology required to describe the proposed approach. This formalism is used to identify suitable deployment scenarios and provide a theoretical reasoning to support the validity of the method. We also discuss the benefits arising from applying the presented approach in existing navigation stacks. Finally, we report the results and quantitative analysis of multiple experiments in various deployment scenarios. 2 Related Work A substantial body of research focuses on detecting and following a specific visual cue. In these applications, the robot can exploit the consistent structure of the environment and follow a certain feature, such as a road, wall, or line. Historically, these systems used single-purpose detectors that relied on a distinct colour [10,11] or shape [12] of a given feature. Some approaches also employ pre-trained segmentation models [13]. These methods are often tied to use in a specific domain and are not approached as a general framework for arbitrary visual cues. More recent approaches can use weakly supervised or reinforcement learning to follow the path. However, these methods usually require large datasets [9,14], simulated environments [15,16], time-consuming training, and powerful hardware. In contrast, the VT&R methods are based on one-shot evaluation and can follow arbitrary visual features. VTAG directly extends VT&R to navigate the robot along the unseen paths, containing a consistent visual cue. The method presented in this paper extends a type of VT&R, often called appearance-based or bearing-only [4]. This type of VT&R simplifies the visual part of the framework. Under the assumptions presented in [5], it is not necessary to obtain a full 6DoF transformation between live and map images. Instead, it is sufficient to obtain only a horizontal displacement between these two images [17,2]. Obtaining a precise and reliable 6DoF transformation between two images requires multiple keypoint correspondences, whereas, in theory, the horizontal displacement estimation can be done with a single correspondence. This horizontal displacement is then used to correct the heading of the ground robot and Rapid Deployment of Visual Path Following via Average Representation 3 steer it towards the desired trajectory. [18] shows that it is possible to replace keypoint matching schemes with the crosscorrelation operator. There are other VT&R methods [19, 20] that rely on consistent metric maps and are less related to the method presented in this article. When we aim for long-term autonomous deployment, the robot has to solve unexpected situations. Primarily, it must be able to avoid collisions during navigation. This is typically addressed by reactive behaviours at the lower level [21, 22]. Another issue is changing the appearance of the environment, which can cause a malfunction of the visual part of a navigation system. At first, researchers focused on handcrafting descriptors robust to variations in point of view and illumination [23,24,25]. With the boom of machine learning in vision, it became apparent that the feature descriptor must be trained rather than hand-crafted [26,27]. Especially in the case of outdoor environments, it is challenging to create a robust descriptor without learning from the data due to common dynamics, e.g. seasonal, day&night changes [28, 29,30,31,32]. One of the promising areas of research that can produce a useful representation of input data is contrastive learning. These representations exist in a latent space created during the training process. The result of the training process is typically a function with learned parameters that transform the input into a latent space where similar data points are located close together. The similarity definition is usually taskspecific. Models using contrastive learning are commonly referred to as Siamese because they consist of two branches with shared model parameters and different inputs [33]. The Siamese networks are suitable for robot localisation because they are explicitly designed for one-shot evaluation. A popular application of a fully convolutional Siamese neural network is tracking objects [34]. This training method creates a spatially consistent representation of an image that can be used to track an object extracted in a one-shot manner. Further research proved that altering this learning method can create a robust feature-matching pipeline. This research refers to the descriptor as a hypercolumn, and the image representation is a dense set of spatially consistent hypercolumns [35]. The major disadvantage of these methods is that they require annotated datasets. The hypercolumn method exploits auto-annotation based on the structure from motion methods; thus, correspondences still need to be obtained before training. Labelling is a major obstacle when working with large datasets. Self-supervised methods are a promising approach to avoid extensive labelling. Many of these architectures are built on contrastive learning approaches [36, 37]. They are not learning similarities using annotation, but instead using a proxy task for the training. The first version of VTAG builds on a self-supervised learning paradigm and was introduced in [9]. It exploits the dense representation matching pipeline [38] and the appearancebased approach used in VT&R navigation [2,4]. The goal of the VTAG framework is to automatically extract distinct visual cues typical for the operational environment and use them for path following. Automatic feature extraction is achieved by collecting a dataset with a consistent position of visual cues in the image space. The dataset is used to train a neural network that produces dense image representations, which contain robust descriptors of a visual cue. The main problem is that the time saved by eliminating the need to map the whole area is offset by the optimisation of the neural network. Moreover, to fine-tune the neural network, collecting a large number (thousands) of images is still necessary. Environment database edge Topological planner Node detection VT&G or VT&R Safety layer Collisions Traversability Robot avg. edge repr. or full edge map steering velocity safe steering and velocity live image odometry, gps, etc. Goal Topology Edgesmaps or avg. repr topomap Fig. 2: Standard navigation stack for appearance based VT&R methods. VTAG can replace maps of some edges in the topological map. The parts of the stack studied in this paper are highlighted in red. The established navigation stacks exploiting the VT&R methods usually rely on non-metric topological maps. The nodes are places important for the robot application scenario, and the edges are traversable paths. The edges are manually mapped to collect the images, which are used by the VT&R to perform the autonomous navigation. The VTAG fits into this paradigm and can be used to traverse the unmapped edges. The purpose of VT&R and VTAG is to provide a simple and robust high-level method to steer the robot along the planned path. It does not provide any traversability estimates or collision avoidance. Lower-level behaviours of the navigation stack must address these problems. The diagram showing such a navigation stack is shown in Figure 2. This approach is called navigation without localisation. In practice, the robot does not need to estimate its position with high precision. Usually, it is sufficient to reach the desired position within a certain distance and then activate task-specific behaviour, which handles the precise control. 3 Method Description This section provides a detailed description of the method presented. First, formal definitions must be made to allow for the description of the framework. All the notation used 4 Zdenˇ ek Rozsypálek∗1et al. for description of the method is summarised in Table 4. The goal is to follow a long path with consistent visual cues (fence, line) or structure (sidewalk, corridor). We call the path that fulfils this property a desired path. Traditionally, a map of the entire desired path must be created for a reliable traversal. Mapping long paths can be undesirable for various reasons, such as time and memory requirements. Instead of the time-consuming mapping, we only collect data over a short subsection of the desired path. We refer to this subsection as training path. The presented version of VTAG consists of three steps: 1. Manually traverse the training path (subsection of the desired path). 2. Transform the training images to the latent space and create the average representation. 3. Use the average representation to navigate the robot through the desired path. In the following subsections, we describe the visual model used in this paper, we elaborate on each of the three deployment steps, and we discuss the benefits arising from using the average representation. 3.1 VTAG Image Processing Model This paper uses the latent space of a publicly available model trained and presented in previous work [38]. Previous research shows that it is possible to use a well-designed proxy task to train a convolutional Siamese neural network with a latent space suitable for horizontal displacement estimation. The proxy task used to train the model is to estimate the horizontal coordinate of an image cutout. The training scheme is similar to [34], but emphasises the application for ground mobile robots. Data pairs are two images with the same position and heading, but taken under different conditions (day&night, seasonal) to make the latent space robust to environmental dynamics and avoid neural network collapse. For inference, the images are not cropped, but one is extended by circular padding to evaluate all possible horizontal shifts exhaustively. The estimated horizontal shift between the map and live images provides a steering correction, navigating the robot along the desired path. In other words, the robot needs to adjust its heading to ensure a perfect horizontal overlap of the map and live images. Following this behaviour leads to convergence towards the mapped trajectory. The evaluation process is defined as:  h=f(IA)⋆ P (f(IB)),(1) where Iis the notion for input images, and fis the trained convolutional neural network (CNN). The CNN output is the image representation, which can be extended by horizontal circular padding Pand cross-correlated (⋆) with other representations. Since both representations have the same height, the output vector  hcontains one value for every possible horizontal displacement between images. The architecture of CNN is described in Table 1. All convolutional filters have size 3×3. The trained model is taken directly from the previous work [38]. Table 1: CNN architecture Layer Elements Shape CH ×H×W Input Image 3×320 ×512 Layer 1 Conv, BN, ReLu, Max Pool 16 ×160 ×256 Layer 2 Conv, BN, ReLu, Max Pool 64 ×80 ×128 Layer 3 Conv, BN, ReLu, Max Pool 256 ×40 ×64 Layer 4 Conv, BN, ReLu, Max Pool 512 ×20 ×64 Output Conv 16 ×6×64 3.2 Mapping Requirements We refer to the image collection process over the training path as mapping. The result of the mapping is a representation of visual features relevant for navigation. These must be consistently present across the collected images, while irrelevant features must exhibit significant variation and diversity. Therefore, the training path must meet the following conditions: 1. Training path shares consistent visual features with the unseen desired path. 2. The visual features are located at a consistent position in the image space of training images. 3. The training images are diverse enough to distinguish navigation-relevant and navigation-irrelevant features. The requirement 1 limits VTAG to usage in repetitive environments. When there is no consistent visual cue along the whole desired path, using an aggregated representation for reliable traversal is impossible. Next, requirement 2 places a constraint on the precision of image collection. Usually, the goal is to maintain a certain distance from a visual cue during autonomous navigation. For example, keep the robot in the middle of the sidewalk or one meter left of the fence. This desired goal must be met during data collection. Otherwise, the visual feature is not located at a consistent position in the image space, resulting in a noisy map representation and unreliable navigation. From our experience, it is easy to manually control the robot and obtain a useful average representation. Finally, requirement 3 is introduced to ensure that the only consistent feature in the training images is similar to the desired path. It is necessary that the only consistent visual feature in the training images belongs to the chosen visual Rapid Deployment of Visual Path Following via Average Representation 5 1 2 3 4 5 6 1 9 17 25 33 41 49 57 64 Fig. 3: Hypercolumn cells overlaid over the image. Each cell in Ris represented by a descriptor q with coordinates x, y. cue. Otherwise, the created representation can then navigate the robot towards the undesired features (i.e. background) instead of following a given visual cue. We collect images from both the front and rear cameras to address this issue. The images taken by the rear camera are horizontally flipped to comply with the requirement 2. This extends the diversity of training images and mitigates the influence of the background. Alternatively, it is possible to use a longer training path, which also reduces the requirement for data collection precision. The minimal required length of the training path is heavily influenced by the spatial properties of the environment. 3.3 Average Representation All processing is performed in a spatially consistent latent space, which is robust to the changing appearance of the scene. The representation Rof the image Iis obtained as R=f(I), where fis a CNN from [38], previously discussed in this section. CNN transforms the input image of size 512 ×320 ×3into a representation of size 64 ×6×16. To comply with established terminology, we refer to each spatial vector as a hypercolumn [35]. Each hypercolumn encodes the image pixels at specific image coordinates. In our case, the hypercolumns q are vectors of size 16 and the representation of width wand height hcan be written as: R=   q1,1. . . qw,1 . . ..... . . q1,h . . . qw,h   .(2) The visualisation of the receptive fields of individual hypercolumns is shown in Figure 3. For our purposes, the representation is significantly denser in the x-axis to allow a precise horizontal matching. We can think of the desired path as a set of consecutive images that can be encountered during its traversal. The representations are obtained by passing the images through the CNN. We denote a set of all representations along the desired path as Rd. The problem is that during data collection, we do not traverse the whole desired path; we only traverse a representative subsection. The set of Ttraining representations collected during the mapping is denoted as: Rt={Rt 1, Rt 2, . . . , Rt T}.(3) Finally, we define a set of all possible existing representations as R. In theory, for a reliable traversal, we need to find an optimal representation R∗. This optimal representation should maximise the response (in our case, the cross-correlation) to all representations in Rdand minimise the response to all existing representations not present in the desired path R\Rd. Formalized, we want R∗to have following properties: Maximize: X Rd∈Rd (R∗⋆ Rd)(4) Minimize: X R∈R\Rd (R∗⋆ R).(5) The representation tensors are the same size; thus, the crosscorrelation outputs a scalar. In practice, this is not a tractable optimisation task. First, given the nature of our use case, the whole set Rdis not available. We only have information about the desired path from the sample set Rt. Secondly, the set of all existing representations Ris extremely large. The first optimization task (4) is addressed by the mapping requirements 1 and 2. All representations in Rdmust contain a feature (hypercolumn) qx,y, which is also present in the representations from Rtat the same position x, y. When this requirement is fulfilled, the visual cue is consistent with the target path and present in the training images. The second optimisation task( 5) is addressed by the mapping requirement 3. We use the robot rear camera with a horizontally flipped output to collect as diverse Rtas possible. When all three requirements are met, it is possible to use representations from Rtto obtain an approximation of R∗. Our approach exploits two important properties of the latent space crafted in previous work. Firstly, the CNN output is robust to various disturbances, such as sensor noise and environmental dynamics. Secondly, the cross-correlation of representations directly estimates the log-likelihood of similarity between the two input images: logL(Ia, Ib) = f(Ia)⋆ f(Ib).(6) Similarity in the path following task means that the images are taken at the same place with the same heading. The neural network used is trained directly to produce a latent space fulfilling both conditions as shown in (1). 6 Zdenˇ ek Rozsypálek∗1et al. Considering the training images as independent observations, we can use the maximum likelihood estimate for the live image Ilto be a good match with all training representations: logL(Rt|Il) = 1 T T X k=1 logL(Il, It k),(7) where Tis the number of training images. The latent space is crafted to compute the log-likelihood of similarities via cross-correlation as in (6). Thus, the likelihood estimate can be rewritten as follows. logL(Rt|Il) = 1 T T X k=1 Rl⋆ Rt k.(8) The cross-correlation is a linear operation, and the live representation can be moved outside the sum. logL(Rt|Il) = Rl⋆1 T N X k=1 Rt k.(9) In this form, we can already see that the average map representation naturally arises from the maximum likelihood estimate. We define the average representation Ravg as: Ravg =1 T T X k=1 Rt k.(10) Finally, when the average representation is plugged into the maximum likelihood estimate, we end up with the final form: logL(Rt|Il) = Rl⋆ Ravg.(11) The experiments show that the average representation can be a good approximation of the optimal representation for the desired path traversal R∗≈Ravg when the mapping requirements are met. Intuitively, the average representation preserves the features consistent in the training images, and the inconsistent parts of the images yield a noisy feature that does not match any real-world objects. Note that by averaging the feature, we do not want to estimate its expected value. That would require the features to be normally distributed in the latent space, which is not the case. We sometimes refer to features as "similar", but our matching scheme does not evaluate the distance between the features. The "similarity" rather means that the correlation of two features yields a high value. Thus, using the average representation is agnostic to the feature distribution. Ultimately, the choice of a linear matching scheme allows us to create a simple aggregated representation of the training path. Standard non-linear matching schemes such as L2 or cosine distance would not allow moving live images outside the sum in (9). The presented derivation shows that using average representation with a linear matching scheme is equivalent to evaluating the live image against all the training images. 3.4 Navigation The navigation step relies on horizontal image displacement estimation via cross-correlation as in (1). The robot should start in a position where the consistent feature is visible. Then, a constant forward velocity is set, and only the heading rotation velocity is controlled by the VTAG. When the live image representation Rland the map representation Ravg overlap perfectly (there is no displacement), the robot traverses the trajectory well, and no heading correction is needed. This is usually not the case in practice, and corrections are necessary in cases where the robot is misplaced, poorly controlled, or the desired path is curved. The heading correction ωapplied to steer the robot through the desired path is calculated as:  h=P(Ravg)⋆ Rl,(12) ∆p =argmax ( h)−wr 2,(13) ω=k·∆p, (14) where Pis a circular padding added to the representation in the horizontal dimension. We use the padding of size wr 2, where wris the width of the representation. The index of the maximal bin in the vector  his used to obtain the horizontal displacement ∆p. Finally, (14) is used to obtain the heading correction ωregulated by a tunable proportional constant kthat can adjust the aggressivity of the control. Note that the distance of visual features limits the convergence speed and can not be easily improved by tuning or altering the regulator. The regulator controls only the robot’s heading, where reference is the direction of the followed features. Convergence towards the desired trajectory is a phenomenon emerging from this strategy and cannot be directly controlled. The properties of appearance-based methods are widely discussed in previous research [4,5]. 3.5 Benefits of Average Representation This subsection highlights features that naturally arise from using the average representation. These features can be beneficial in multiple autonomous deployment scenarios and do not require changes to the standard navigation stack. 3.5.1 Real-time Feature Engineering and Interpretability The major benefit of our approach is that it can be deployed in real-time for an arbitrary consistent visual cue. The av- Rapid Deployment of Visual Path Following via Average Representation 7 erage representation can be accumulated during teleoperated traversal and immediately evaluated with live histogram plotting. This online feature selection can reduce personhours, which have to be spent by computer vision engineers to design the detector of the visual cue. Such a task often requires highly skilled workers, and reducing their workload can result in cheaper deployments. Moreover, in some scenarios, it is not obvious how the one-purpose detector should be designed. For example, traversing the sandy pathway (as in Section 4.3) can be implemented in multiple different ways, e.g. detecting the boundaries of the pathway, segmenting the sandy surface, and many more. Our approach removes these design choices from the process by exploiting the entire image and extracting the most consistent and reliable features. The proposed simplification of the workflow is presented in Figure 4. Live representation Live representation General neural network Average representation Navigation Trained NN Neural network training/validation Dataset repr. Live repr. Fine-tuned neural network Navigation VTG via average representation (ours) VTG Map images Dataset Camera Representation sum + 1/T   Map repr. Dataset of representations Server Navigation Mapping Camera Fig. 4: Diagram presenting different workflows of previous work [9] and presented method. The average representation enables significant simplification of the deployment process. Using a server to train a new neural network for every visual cue is no longer necessary. Using our method as a black box can make it difficult to interpret the outputs of the created detector. It is always beneficial to have insight into the decision-making of the system so that the engineer can predict the robot’s behaviour in various edge cases. The spatial consistency representation makes it possible to evaluate how different parts of the image contribute to the output. The likelihood of all possible displacements (12) can be rewritten using the summation notation as:  h[d] = c X i=1 h X j=1 w X k=1 P(Ravg)[i, j, k +d]·Rl[i, j, k],(15) where c, h, w are the representation’s channels, height and width, respectively. The matching scheme uses summation over the spatial dimensions to calculate the similarity of the whole representation. When we want to see the contribution of a hypercolumn (bin) representing a section of the image, we can set the displacement das a fixed value and omit the summation over the spatial dimensions. Final contribution matrix Cis computed as: C[j, k] = c X i=1 P(Ravg)[i, j, k +d]·Rl[i, j, k],(16) where the indices j=y, k =xnow correspond to the spatial position of the hypercolumn qx,y. Note that dis fixed at the value that produces the maximal likelihood in (15). Contributions directly quantify the similarity of individual hypercolumns. The visualisation of this approach is shown in Figure 10. The CNN significantly squeezes the height dimension, which makes the contribution less precise in the vertical axis. 3.5.2 Semantic Localisation Another benefit of our method is based on the fact that the Siamese networks are designed for one-shot evaluation. The cross-correlation outputs higher values when the two representations are well-matched. The likelihood estimate can be reversed so that we can find the best set of training representations that matches the current live image logL(Il|Rt). This property extends the versatility to the entire framework. The live representation Rlcan be cross-correlated with multiple average representations. The representation Ravg that produces the highest likelihood can be chosen to control the robot. This simple concept can extend the framework’s usability, e.g., in multi-experience-based navigations like [39,40]. The feasibility of this approach was already demonstrated in our previous work [41]. We also used semantic localisation in our experiments, as shown in Figures 12 and 15. Note that semantic localisation in its current form can not directly estimate a precise metric position of the robot. 4 Experiments Five experiments were conducted to evaluate the capabilities of the improved VTAG framework. Each experiment 8 Zdenˇ ek Rozsypálek∗1et al. Fig. 5: Photos of the robots used in the experiments. takes place in a different type of environment. The mobile robot used for experiments 1 and 2 is the Boston Dynamics Spot. Experiments 3, 4 and 5 were conducted with the Clearpath Jackal. Both robots have Basler Ace 2 a2A192051 cameras facing forward and backwards. The Jackal robot has one more camera facing to the left. All cameras stream 1080p video at 40FPS, resized to 512 ×320 for processing. The Spot robot uses an Intel NUC i7 9th Gen, and the Jackal robot uses the NVIDIA Jetson AGX Orin. Additionally, the ground truth for experiments 1 and 2 is provided by the LiDAR Ouster OS1. The robotic platforms can be seen in Figure 5. The processing speed and comparison to the previous work are presented in Table 2. Table 2: Methods comparison VTG [9] OUR CNN size (params) 1.4M CNN forward pass (NUC) 50ms CNN forward pass (ORIN) 15ms Required image num. thousands tens Representation building hours/days real-time Descriptor size 16MB 100KB The Spot robot uses proprietary software for collision avoidance and traversability estimation. It is impossible to turn off these low-level behaviours for safety reasons enforced by the manufacturer. Our framework provides a higher level of control to navigate the robot along the desired path. 4.1 Experiment 1: Corridor Traversal In the first experiment, we show the convergence properties and applicability in an indoor environment. The goal is to use VTAG to keep the robot centred in the middle while walking forward through a corridor. The experiment is conducted in the FEE CTU building in Prague on the Charles Square campus. The building has most corridors with roughly similar dimensions but different appearances caused by wall colours, furniture, and window placements. In the experiment, we create a map representation based on the traversal through the centre of one corridor, and test it in two other corridors. The mapping traversal is 15m long, and we collect two images (front and rear) per meter, so the average representation of the campus corridor consists of 30 images in total. All corridors traversed during the experiment are shown in Figure 6. Using the methodology proposed in [4,5,41], we place the robot at the beginning of each corridor with the heading and lateral displacement of ≈1 m from the corridor centre. Then, the system autonomously navigates the robot through the corridor. According to the model specified in [4,5], the robot’s heading is corrected almost instantaneously, and lateral displacement (from the centre of the corridor) gradually converges to zero as the robot traverses the corridor. As noted above, the convergence speed is mainly constrained by the structure of the environment. Fig. 6: The two top images are examples taken by the robot in the mapped corridor 2. The two bottom images are from not mapped corridors 1 and 3. LiDAR data were recorded during the traversals to quantify the convergence. We use RANSAC [42] to find two collinear lines 3 m apart representing the corridor’s walls. When the position of the walls is estimated, it is possible to calculate the robot’s lateral displacement from the corridor’s centre. Since the corridors are not exactly the same width, and the LiDAR is placed on top of the walking robot, the precision of the ground truth is about 0.1m. Additionally, the navigation is influenced by the limited precision of the teleoperated image collection. The displacements from the centre of the corridor during the traversals are shown in Figure 7. The mean absolute error of corridor centre traversal is 0.07 ±0.04m. This value is computed from measurements collected after 10 m of traverse, where the robot has already corrected the introduced displacement. Rapid Deployment of Visual Path Following via Average Representation 9 0 5 10 15 20 25 30 Traversed distance [m] 0.0 0.2 0.4 0.6 0.8 1.0 Lateral displacement [m] Convergence to corridor center Corridor 1 Corridor 2 Corridor 3 Fig. 7: Convergence to the centre of the various corridors using one corridor representation. 4.2 Experiment 2: Rail Following The second experiment is conducted in an urban outdoor environment along the banks of the Vltava River in Prague. The experiment investigates whether it is possible to use the proposed method to navigate over non-trivial paths while performing semantic localisation. The goal is to keep a defined distance from the railing and traverse a long path along the river. The sidewalk consists of different surfaces and is lined with a railing. The appearance of the railing varies slightly throughout the desired path. We mapped around 20 m of the path along the railing and traversed a trajectory of 500 m back and forth, resulting in over a kilometre-long traversal with a single manual turn by 180 deg at the end. The desired path consists of two distinct sections. One-half closely resembles the training path with a bicycle lane, but the second-half has a different surface and railing. The chosen place is relatively busy, and the navigation stack has to deal with pedestrians and cyclists in the camera view. Furthermore, the desired path has a significant curvature, and there is a challenging situation B where the railing diverges from the bike lane. The whole trajectory is visualised in Figure 8. Fig. 8: Trajectory visualisation. The image is only illustrative and does not represent any measured data. The aerial imagery is taken from mapy.com. The places highlighted in blue refer to challenging situations A and B during traversal. Training images are collected by manually controlling the robot at a fixed distance from the railing (0.9 m). The images were taken once per meter and used to create the average representation. When the representation is created, we take the robot to the start of the desired path and let the framework perform the traversal. The beginning of the desired path looks similar to the training trajectory except for the background. However, after ≈250 m, the appearance changes significantly. Figure 9 shows training images and images from challenging sections of the desired path. Fig. 9: The two top images are samples of training images. The rear camera image must be horizontally flipped to create a consistent representation. The two bottom images show sections of the desired path with different appearances. In Figure 11, we report the ground truth distance to the railing measured by LiDAR and the estimated horizontal displacement used for heading correction throughout the traversal. The robot can successfully keep the defined distance with two exceptions. Firstly, after traversing ≈30 m (situation A), collision avoidance is activated to avoid a pedestrian. The robot deviates from the path and then autonomously converges back. Secondly, the robot enters the challenging section at a distance of ≈250 m. The system chooses to follow the bike lane instead of the railing, resulting in a smoother trajectory (situation B). Note that the railing is not a dense surface, adding noise to the ground truth estimate. The second half of the desired path also contains gaps in the railing, as shown in Figure 9. These gaps further degrade the ground truth quality and are visible in the reported lateral displacements. In this experiment, we also showcase the additional features of the framework. Firstly, semantic localisation is used to estimate the rotation of the robot. We have two representations, one with the railing on the left-hand side and one horizontally flipped. The robot evaluates both representations simultaneously and chooses the best-matching representation for heading adjustments. This concept allows us to traverse