LAIF: Learning by demonstration through Active Incremental data Fusion of task observations

LAIF: Learning by demonstration through Active Incremental data Fusion of task observations Dimitrios Papageorgiou1,2, Nikolaos Efstathopoulos1, Yann Angus3, Nikolaos Kounalakis4 This paper is a pre-print version presented and included in the proceeding of IEEE-IST 2025 conference. DOI: 10.1109/IST66504.2025.11268388 . Abstract—As William Arthur Ward once said: ”Curiosity is the wick in the candle of learning”. In this work, inspired by the curiosity that drives human beings, we propose an active perception framework aiming towards the incremental learning through visual observations of multiple demonstrations, performed by a human. The proposed framework, considers an active observer, i.e. a sensor with the ability to move in Cartesian space, which acts in order to maximize the information gathered towards the modelling of the observed motion. For the the encoding of the human action, a Dynamic Movement Primitives (DMP) model is utilized. In the core of the method, a Kalmanfilter-inspired data fusion mechanism is employed that exploits the knowledge of the trained DMP model, and accounts for the uncertainty of the current knowledge and the measurement uncertainty, in an iterative manner. The proposed method is tested using a UR5e robotic manipulator with an eye-in-hand ZED 2 camera in two different scenarios, involving the motion of the human hand along a curve, considering the existence of occlusions within the field of view of the camera, as well as anocclusion-free case. The proposed method is theoretically proven to minimize the uncertainty in each repetition and its performance is demonstrated trough the results of the experimental evaluation. Index Terms—Active Vision, Data fusion, Imitation Learning. I. INTRODUCTION Learning by demonstration (LbD) has gained the researchers’ attention worldwide [1], as, in contrast to the traditional explicit programming that is commonly used in industry, it provides an intuitive means for robot programming. According to LbD, the robot is able to learn a kinematic behavior based on a set of demonstrated motions that can be gathered from any perceptual system, e.g. through visual observations [2], [3], or from proprioception [4], [5], i.e. by the robot’s own internal sensors. Although LbD through visual observations is widely investigated in current literature, mainly passive observations are considered. In other words, most of the works assume a passive observer that does not posses the ability of moving in space for maximizing the information 1Dept. of Electrical and computer Engineering, Hellenic Mediterranean University, Heraklion, Greece. [email protected] 2Institute of Computer Science, Foundation for Research and Technology - Hellas, Heraklion, Greece. 3Dept. of Electrical and Industrial Computer Engineering, Institute of Technology of Rennes, France. 4Dept. of Mechanical Engineering, Hellenic Mediterranean University, Heraklion, Greece. The research is funded by CARPOS project, carried out within the framework of the National Recovery and Resilience Plan Greece 2.0, funded by the European Union - NextGenerationEU (Implementation Body: HFRI). Project Number: 16523. gathered. That makes the learning procedure prone to the uncertainty involved in the sensor measurement, which may occur, among others, due to possible occlusions, measurement noise or in general erroneous position estimation (e.g. error in depth estimation). Most LbD frameworks in literature utilize Dynamical Systems (DS) for encoding the characteristics of the kinematic behavior. From a modelling perspective, LbD can be seen as the problem of finding the parameters of the dynamical system that yield a kinematic behavior which optimally mimics the demonstration dataset, given the same initial conditions. Dynamical systems, as encoding and motion generation mechanisms, provide generalization capabilities, as well as the ability to on-line adapt to dynamic changes. The most common DSs utilized in LbD, in the literature, are the Dynamic Movement Primitives (DMPs) [6] and the Gaussian Mixture Models (GMMs) [7]. Data Fusion aims towards the optimal exploitation of multiple data sources for yielding accurate and reliable information about a perceived value. Kalman filtering, introduced in early 60s by Rudolf E. K´ alm´ an [8], and its more recent variants, namely the Extended Kalman Filter (EKF) and the Unscented Kalman Filter (UKF) [9], are considered to be the most commonly utilized data fusion methods for estimating the state of a system, based on an already known dynamic model of the system and considering a measurement procedure. The uncertainty involved in both the modelling and the measurement are considered to have known statistical characteristics, i.e. a known covariance. The state estimation is essentially performed by fusing the prediction based on the model, with the measured output of the system. Although EKF, UFK or similar data fusion methods were previously combined with DS-based motion generation mechanisms, such as in [10] that aims towards achieving an effective human-robot collaborative object transfer, they were not utilized previously for tackling the problem of active perception for LbD. The notion of Active Perception was introduced by Aloimonos et al. [11], which involves an active observer able of acting in order to maximize the amount of information gathered while perceiving a phenomenon, e.g. a robot with an eye-in-hand camera [3], [12]. Active Perception is widely studied in literature, including industrial [13], [14] and agricultural applications [12], [15], as well as in manipulation [16]. However, to the best of the authors’ knowledge, its use to minimize the uncertainty aiming towards increasing the performance of LbD, is not considered previously, given the availability of an active observer. Furthermore, the use of Active Perception for tackling the problem of LbD was only considered in [3], in which however the learning was not performed incrementally, thus requiring high memory capacity for storing the data of each repetition, and no occlusions of the point of interest were considered. In this work, we propose a curiosity-driven LbD methodology, which is based on active observations of recurring demonstrations of the same task. The proposed method considers an active observer (i.e. an observer that can change its position and orientation in Cartesian space), employs a Kalman-filter-like data fusion mechanism for the estimation of the observed system’s state and utilizes Dynamic Movement Primitives for the modelling of it in each repetition. The aim of the active observer is to optimally move in order to minimize the uncertainty of the estimate provided by the data fusion mechanism, in real-time, in each iteration, aiming to the optimal encoding/modelling of the observed action. Towards this direction, a control signal is proposed that is able to move the sensor in order to optimally perceive the position of the point of interest. II. PROBLEM DESCRIPTION AND OBJECTIVE Let us consider a Point Of Interest (POI), e.g. human fingertip, whose motion (position in time) is required to be encoded/modelled by the system during the execution of the task. We consider the availability of recurring demonstrations of the same task, e.g. multiple repetitions. Let p(t)∈R3 be the position of the POI with respect to the inertial frame {0}. Further, consider a sensor (or perception system), e.g. camera, which is attached to a mechanism able of moving in Cartesian space. Let {P}be the frame of the sensor and vp≜[˙ p⊺ pω⊺ p]⊺∈R6its generalized velocity with respect to {0}, with pp∈R3being its position with respect to {0}and ωp∈R3its angular velocity. We consider the case in which one can command the value of vp(t)in real-time, while a low-level control scheme is responsible of accurately tracking it1. Let ˆ p=p+ε∈R3be the measured (by the sensor) position of the POI with respect to {0}, which is assumed to involve a measurement noise/error ε∈R3. The measurement noise is considered to be a random variable having a normal probability distribution, i.e. ε∼ N(ˆ p,Σ), where Σ∈S3 ++ is the variance-covariance matrix of εwith respect to {0}, with SN ++ being the set of N×Npositive definite matrices. The covariance matrix is assumed to be a known function of the orientation of the sensor and the position of the POI, being characterized by the following ellipsoid surface: S(Σ)≜{y∈R3:y⊺Σ−1y= 1}. Let the pose of the sensor be zp≜[p⊺ pq⊺ p]⊺∈R3×S3with qp∈S3being the orientation of {P}with respect to {0}in a unit quaternion form, with S3⊂R4being the set of unit vectors in R4. Our aim is to design a framework able of acquiring and exploiting information about a recurring phenomenon, namely 1This is the case in many modern robotic systems, such as mobile robotic vehicles and robotic manipulators. Fig. 1: The proposed framework at each repetition. the human demonstration of the same task, by incrementally fusing the data gathered in each repetition with the knowledge accumulated from the previous repetitions. Towards a better definition of the problem, let iˆ p∗(t)∈R3be the total (accumulative) estimate at the end of the i-th repetition of the phenomenon/demonstration provided by the proposed system and iP(t)∈S3 ++ its related covariance. Our objective is to ensure that the volume enclosed by the ellipsoid S(iP(t)), which is proportional to det iP∈R≥0, is minimized for all t∈[0, T]in each incremental repetition of the task, i.e. from ito i+ 1, where T∈R>0is the total duration of the task demonstration. III. THE ACTIVE INCREMENTAL DATA FUSION FRAMEWORK Let iˆ p∈R3be the measured by the sensor position of the POI during the i-th repetition of the task and iˆ p∗(t)∈R3 be the total estimate provided by the proposed method at the end of the i-th repetition. The main idea of the proposed method is: a) to incrementally (at each repetition of the task demonstration) use the previous model of the task, which is provided by modelling based on the previous estimate, and b) fuse the new incoming measurements with the ”prediction” from the model for yielding the new total estimate, and so forth. The total number of repetitions required for a reliable modelling can be assessed by the covariance of the estimate. Towards this direction, the model utilized during the (i+ 1)- th repetition, is based and encodes the total estimate of the previous repetitions, i.e. iˆ p∗(t)for all t∈[0, T]. For the modelling of the system, the Dynamic Movement Primitives (DMP) model is utilized which is briefly presented below. Moreover, for optimizing the point of view of the sensor, an active perception control signal is proposed that moves the sensor in order to achieve minimization of the uncertainty involved in the data fusion, which is based on both the modelling and the measurement uncertainty. The complete framework is conceptually depicted in Fig. 1. Remark 1. Note that the data fusion mechanism selected is the covariance-weighted least square solution among the model’s prediction and the measurement, which is very close to EKF. The selection of this EKF variant, as compared to the original EKF, is based on extensive comparison through simulations, which showed that, in our case, it yields better results, due to the fact that original EKF is highly dependent on the initial parameter selection, i.e. the initial uncertainty is propagated to the whole estimate and it is characterized by a transient. Furthermore, this specific variant is significantly computationally lighter, as the on-line calculation of the Jacobian of the DMP is not required. A. Motion encoding at the end of the (i−1)-th repetition At the end of the (i−1)-th repetition, the signal (i−1) ˆ p∗(t) has been calculated for all t∈[0, T]. Based on this estimate a DMP model is trained, which is, in turn, used by the data fusion mechanism for the ”prediction” during the ith repetition. The DMP model is a non linear dynamical system of the form: ˙ x=f(x;W), where f(x) : R7→R7, x≜[x⊺ 1x⊺ 2x3]⊺∈R7is the complete state of the system, with x1≜p∈R3being the part of the state representing the position of POI (encoding/modelling (i−1) ˆ p∗(t)), x2∈R3its velocity and x3∈R>0is an auxiliary variable, called “phase variable”, which replaces time in order for the system to be autonomous. The most important parameters are W∈R3×M, which constitutes the “weights” that are adjusted (during the “training” phase) in order for the evolution of the dynamic system to mimic as much as possible the training dataset, with M∈Nbeing the predefined number of kernels used. More details about the DMP model can be found in [6]. B. State estimation through data fusion at the i-th repetition The measurement provided by the sensor is fused with the “prediction” from the DMP model by employing a covarianceweighted mean (weighted least square solution) of the data. The rationale behind this selection is to give more importance to the measurement towards the directions in which it is more reliable, as compared to the reliability of the prediction provided by the current modelling, in each iteration, and vice versa. Let w(t)∈R3be the prediction noise, i.e. the error between the actual position of the POI and the position prediction provided by the DMP model. Given that the DMP model is trained based on the state estimate provided at the end of the previous repetition, it incorporates the uncertainty of this estimate. Therefore, assuming a sufficiently accurate endcoding, w(t)is considered to be a random variable, having a normal distribution with covariance iQk∈S7 ++, which is equal to the uncertainty of the total estimate after at end of the i−1repetition (i.e. the previous total estimate), or in other words, we set: iQ(t) = (i−1)P(t),∀t∈[0, T],(1) where (i−1)Pk∈S7 ++ the covariance of the uncertainty of the total estimate during the (i−1)-th repetition, the calculation of which follows. Remark 2. Notice that for i= 1, i.e. during the first repetition, there is no previous demonstrations and therefore one can consider only the linear part of the DMP with a relatively high uncertainty, e.g. iQ=σ2 0I7, with σ0∈R>0a relatively large value. For fusing the position measurement provided by the sensor, i.e. ˆ p(t), with the position prediction provided by the DMP model, which is represented by x1(t), we propose the following covariance weighted mean, as a data fusion mechanism: iˆ p∗(t) = iQ−1(t) + Σ−1(t)−1iQ−1(t)x1(t) + Σ−1(t)ˆ p(t).(2) Notice that (2) is essentially the weighted left pseudoinverse solution (weighted least squares) of the following forward relationship: [x⊺ 1(t)ˆ p⊺(t)]⊺= [I3I3]⊺iˆ p∗(t),considering the following weight matrix W=diag(iQ−1(t),Σ−1(t)), in order to account more for the directions with less uncertainty. The variance-covariance matrix of the total estimate, i.e. iP(t)∈S3 ++, can then be calculated by: iP(t) = iQ−1(t) + Σ−1(t)−1.(3) Proof of (3):. Given that x1(t)is characterized by the modelling uncertainty, having a covariance of iQ(t), and ˆ p(t) by the measurement covariance, i.e. Σ(t)and by taking into account (2), one gets: iP(t) = var(iˆ p∗(t)) = iQ−1+Σ−1−1iQ−1iQiQ−⊺iQ−1+Σ−1−⊺ +iQ−1+Σ−1Σ−1ΣΣ−⊺iQ−1+Σ−1−⊺. (4) Due to the fact that both iQand Σare symmetric positive definite matrices, we have iQ−⊺=iQ−1,Σ−⊺=Σ−1, (iQ−1+Σ−1)−⊺= (iQ−1+Σ−1)−1and therefore, (3) yields from (4). Lemma 1. An ellipsoid S(A)is enclosed (or radially surrounded) by another ellipsoid S(B), where A,B∈S3 ++, if y⊺A−1y>y⊺B−1yfor all y∈R3. The proof of Lemma 1 is provided in Appendix A. Theorem 1. The covariance ellipsoid of the total estimate at the i-th repetition, i.e. SiP(t), is enclosed (i.e. radially surrounded): 1) By the covariance ellipsoid of the (i−1)-th repetition, i.e. S(i−1)P(t). In other words, the uncertainty of the estimate at each repetition will always be less than the uncertainty at the previous repetition, along any direction. 2) By both the covariance ellipsoid of the modelling error, i.e. SiQ(t), and the covariance ellipsoid of the measurement, i.e. S(Σ(t)). In other words, the uncertainty of the total estimate will always be less than both the measurement and the modelling uncertainty, along any direction. Proof. Given that (i−1)P(t) = iQ(t)from (1), S((i−1)P)will constitute the surface for which: y⊺(i−1)P−1y=y⊺iQ−1y= 1,y∈R3,(5) while S(iP)will constitute the surface for which: y⊺iP−1y=y⊺(iQ−1+Σ−1)y= 1,y∈R3.(6) Given that both iQand Σare positive definite matrices, the following will hold: y⊺(iQ−1+Σ−1)y>y⊺iQ−1y.(7) After substituting (5) and (6) into (7), we have: ρ > y⊺(i−1)P−1y, ρ > y⊺Σ−1y, ρ > y⊺iQ−1y,(8) with ρ≜y⊺iP−1y, for all y∈R3. Therefore, according to Lemma 1, the first inequality of (8) proves the first part of Theorem 1, while the second and third inequalities prove the second part of it. Remark 3. Assuming that one can rotate the covariance ellipsoid of the measurement, i.e. S(Σ), by rotating the sensor, the second part of Theorem 1 implies that the uncertainty can be optimally minimized by ensuring that S(Σ)is orthogonal to S(iQ), i.e., the major axis of Σis aligned to the minor axis of iQand vice versa (given that they belong to S3 ++). C. Maximizing information through motion From (3), one can see that iPdepends on Σ(qp), which in turn depends on the orientation of the sensor, i.e. qp∈S3. Therefore, as our aim is to move the sensor such that the covariance of the uncertainty of the estimate is minimized, the following optimization problem has to be solved in realtime during the observation of the demonstration at the ith repetition: J≜minqp∈S3det(iP(qp)).To this aim, we propose the utilization of the following control signal, as a commanded velocity for the sensor: vp≜−kaS(pc−pp) I3J⊺ q(qp)∂det(iP(qp)) ∂qp⊺ ,(9) where pc∈R3defines the constant desired center of rotation, or pivot point, of the sensor’s movement, which is selected by the designer (it could be defined as the center of the scene), S(a) : R3→R3×3is the skew-symmetric matrix mapping of any a∈R3,Jq(qp)∈R4×3is the matrix that maps the angular velocities to unit quaternion rates, for which it holds J⊺ q(qp)Jq(qp) = I3. Note that, the columns of Jq(qp)define an orthonormal base on the tangential plane to the unit quaternion sphere and therefore Jq(qp)J⊺ q(qp)is the projection matrix of any vector in R4to the tangential vector space of S3, i.e. the space of feasible unit quaternion rates. Given the above properties, it is easy to show that J⊺ qa=J⊺ q(JqJ⊺ q)a,∀a∈R4and therefore, the last part of (9) constitutes the projection of ∂V (iPk(qp)) ∂qp⊺ to the feasible, tangential to the unit quaternion sphere, directions. One can find the exact definition of Jqin Appendix A of [17], denoted by the symbol ”JQ”. Furthermore, employing the Jacobi’s formula, the last term of (9) is calculated as: ∂det(iP) ∂qj =∂det(iP) ∂qj =det(iP)·tr iP−1∂iP ∂qj,(10) with ∂iP ∂qj =iQ−1+Σ−1−2Σ−2∂Σ ∂qj ,(11) where qj, with j= 1, ..., 4the elements of qp∈S3⊂R4. D. The complete framework Integrating the above components in one single method, as shown in Fig. 1, the proposed solution can minimize the uncertainty involved in the modelling of the task, within a number of iterations. Regarding the memory capacity required, in each iteration, only the trained DMP model weights and the covariance of the estimate have to be stored, i.e. Wand iPkfor all k∈[0, kN]respectively, where kN∈Nthe total number of samples gathered, as opposed to [3], in which the learning was not performed incrementally. The Python (real experiments) and Matlab (simulation) implementation of the algorithm can be found in the following link, providing also a pseudocode: https://github.com/CSRL-HMU/AIF-LbD. Notice that the computational cost of computing (11) is O(n3)with n= 6, which means that the algorithm is implementable in real-time without high-computation requirements. IV. EXPERIMENTAL VALIDATION A video of the experimental and simulation evaluations, can be found in this link: https://youtu.be/ECnM2dK781w. For the experimental validation, the UR5e robot is utilized, having an in-hand ZED 2 RGB-D camera sensor attached to its tip, while the Point Of Interest is considered to be the index finger of the human. The framework was implemented using Python, while for the forward kinematic and the Jacobian of the robot, the Robotics Toolbox (https://petercorke.com/toolboxes/roboticstoolbox/) was utilized. For identifying the human fingers, the pre-trained “Mediapipe Hands” [18] algorithm was utilized in real time, while for identifying any other objects (possibly occluding the field of view), YOLOv8 (https://docs.ultralytics.com/models/yolov8/) was employed. The latter detection algorithms were running on a separate (from the control loop) thread, in order to ensure that the control cycle, which was set to 2ms, is not affected. The resolution of the camera was selected to be 1280x720, while its frame rate was set to 60 FPS; however, this rate was reduced by the detection algorithms which were running in real time, therefore yielding a new finger estimate approximately every 71ms. It should be noted that when no depth values were provided by the sensor, i.e. a ”NaN” was given, a default preset value was considered for the depth. The estimated covariance of the uncertainty of the camera sensor was set to σ1=σ2= 40 pixels, both on xand yaxis of the pixel space, sourcing mainly from the fingerdetection accuracy, while the covariance of the depth estimate was set to σ3= 0.3m. Given that the intrinsic parameters of the camera are provided by the manufacturer, the covariance of the 3-D position estimate, based on the back-projection pinhole relationship of the camera, is given by: Σ(qp) = 2µRp(qp)diag σ2 1 f2 x (σ2 3+ ˆp2 cx),σ2 2 f2 y (σ2 3+ ˆp2 cy), σ2 3R⊺ p(qp), where µ∈ {0,1}is the binary variable that takes the value ”0” when the POI is not occluded and ”1” if it is, fx, fy∈R (a) The case of fully visible POI. (b) View of the camera (case involving occlusions of POI) Fig. 2: The two cases considered for the experimental validation. the focal lengths of the camera sensor along the xand yaxes respectively, and ˆpcx,ˆpcy ∈Rthe xand ycoordinates of the position estimate of the POI with respect to the camera frame {P}. Notice that the uncertainty is considered to be doubled when the POI is occluded, due to the multiplication with 2µ. Furthermore, notice that the ellipsoid’s volume is minimum when the point is located at the center of the camera, i.e. when pcx =pcy = 0. For the DMP model, we utilized M= 100 kernel functions and a linear canonical system. The gain of the proposed method was selected to be ka= 5 ·104when no occlusions were considered in the scene, and ka= 3 ·104, when considering occlusions. Notice the relatively large value of the gain, due to the fact that the control signal involves its multiplication by the volume of the ellipsoid, which is relatively small (scale of 10−6). For the experiments, the human was instructed to follow a predefined path annotated on a table, in order to use it as a ground-truth for the evaluation, as shown in Fig. 2. No information about this path was provided to the proposed framework. For testing the performance of the proposed framework, we consider two cases. The first case involves the observation of the finger’s motion without considering any occlusions to the scene (Fig. 2a), while for the second case an obstacle (plastic mock-up fruit) was considered to intercept the view of the POI, during a part of its motion, as shown in Fig. 2b, as seen by the camera. For detecting if an occlusion occurs, we compare the depth of the fingertip’s estimate (provided by MediaPipe) to that of the object (identified by YOLOv8), if the finger’s position in the pixel space is within the bounding box of the object2. Lastly, three repetitions of the task were considered, as during the third repetition, the uncertainty of the estimate was deemed already sufficiently minimized. In Fig. 3 the paths of the estimated position of the POI are shown, in each repetition i, for both cases: without and with occlusions. The orientation of the camera during its motion can be inferred by the direction of the Σ-ellipsoid, in each case. Notice the noise of the measurement during the first repetition when no occlusions are considered (Fig. 3a), mainly 2Notice that a simplified rationale was employed for this specific aspect, as the focus of the proposed framework is not towards the development of such detection algorithms due to ”NaN-value spikes” in depth estimate, which however are sufficiently rejected by the proposed method after the second repetition of the demonstration (black solid line in Fig. 3b). Further notice the action of the camera during the second observation (from the ellipsoids in Fig. 3b), which aims towards achieving the orthogonality between Σand iQ. Let us note that the velocity (i.e. its total motion) of the sensor during the third observation was significantly reduced, as the volume of iPwas relatively small. This condition can serve as an indication that no further repetitions are required, i.e. that the system has accumulated sufficient knowledge. In the case in which occlusions are considered, notice the erroneous path segment, at the middle of the motion during the first (i= 1) repetition, associated with a relatively larger uncertainty (Fig. 3e), due to the occlusion of the POI by the obstacle. However, notice that due to the high uncertainty considered automatically by the proposed method, during the 2nd repetition (Fig. 3f) this erroneous segment is smoothly ”replaced” by the corresponding part of the measurement, which did not involve any occlusions, as the pose of the sensor was automatically optimized by the proposed framework. Lastly notice the reduction of the uncertainty achieved due to the motion of the sensor, based on the proposed signal, during the second repetition, in both cases, which is reflected by det2P(t), depicted in Fig. 3d and 3h for each case respectively. V. CONCLUSIONS AND FUTURE RESEARCH In this work, an active perception Learning by Demonstration framework is proposed, considering the availability of recurring demonstrations of the same task and an active observer (active sensor). The proposed framework combines Kalmanfilter-like Data Fusion, Dynamic Movement Primitives and the notion of Active Perception for achieving optimal learning within a couple of repetitions of the task, performed by the human. The experimental results, which consider a robot with an eye-in-hand RGB-D camera and involve occlusions and erroneous depth measurements, demonstrate the minimization of the uncertainty achieved by the proposed method, as well as noise rejection, within a couple of repetitions. Future research directions involve the extension of the method to incorporate multiple active observers and its application to realistic environments for the modeling of Repetitive Dynamic Events (e.g. recurring fire in critical assets). Regarding the considered hardware, future directions involve the utilization of mobile robots, such as quadruped robots, and different sensor modalities, such as thermal cameras. REFERENCES [1] A. Billard, S. Calinon, R. Dillmann, and S. Schaal, Robot Programming by Demonstration. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, pp. 1371–1394. [2] R. Dillmann, “Teaching and learning of robot tasks via observation of human performance,” Robot. Auton. Syst., vol. 47, no. 2, pp. 109–116, 2004. [3] D. Papageorgiou, N. Kounalakis, N. Efstathopoulos, J. Fasoulas, and M. Sfakiotakis, “Activo: An active perception framework for skill transfer through iterative visual observations,” in 2024 32nd Mediterranean Conference on Control and Automation (MED), 2024, pp. 94–100. (a) Repetition i= 1. Case: No occlusions. (b) Repetition i= 2. Case: No occlusions. (c) Repetition i= 3 Case: No occlusions. 0 2 4 6 8 0 1 2 3 4 510-7 (d) det(2P(t)) during Repetition i= 2. Case: No occlusions. (e) Repetition i= 1. Case: With occlusions. (f) Repetition i= 2. Case: With occlusions. (g) Repetition i= 3. Case: With occlusions. 02468 0 2 4 6 8 10 10-6 (h) det(iP(t)) during Repetitions i= 1,2. Case: With occlusions. Fig. 3: Path of the incremental estimate and det(2P(t)), in both cases. Cyan dotted line: ˆ p(measurement), black solid line: iˆ p∗(total incremental estimation), green dotted line: x1(t), red ellipse: S(iQ), blue ellipse: S(Σ), black ellipse: S(iP), yellow dashed line: pr(t) (reference/ground truth position of the POI). Notice that the surface S(1Q(t)) (at i= 1), is not depicted, as for i= 1 it consists of a sphere of a relatively large radius that would distort the presentation. [4] D. Papageorgiou, T. Kastritsi, and Z. Doulgeri, “A passive robot controller aiding human coaching for kinematic behavior modifications,” Robot. Comput.-Integr. Manuf., vol. 61, p. 101824, 2020. [5] D. Papageorgiou, F. Dimeas, T. Kastritsi, and Z. Doulgeri, “Kinesthetic guidance utilizing dmp synchronization and assistive virtual fixtures for progressive automation,” Robotica, vol. 38, no. 10, p. 1824–1841, 2020. [6] A. J. Ijspeert, J. Nakanishi, H. Hoffmann, P. Pastor, and S. Schaal, “Dynamical movement primitives: Learning attractor models for motor behaviors,” Neural Comput., vol. 25, no. 2, pp. 328–373, Feb 2013. [7] S. M. Khansari-Zadeh and A. Billard, “Learning stable nonlinear dynamical systems with gaussian mixture models,” IEEE Transactions on Robotics, vol. 27, no. 5, pp. 943–957, 2011. [8] R. E. Kalman, “A New Approach to Linear Filtering and Prediction Problems,” Journal of Basic Engineering, vol. 82, no. 1, pp. 35–45, 03 1960. [Online]. Available: https://doi.org/10.1115/1.3662552 [9] S. Julier and J. Uhlmann, “Unscented filtering and nonlinear estimation,” Proceedings of the IEEE, vol. 92, no. 3, pp. 401–422, 2004. [10] A. Sidiropoulos, Y. Karayiannidis, and Z. Doulgeri, “Human-robot collaborative object transfer using human motion prediction based on dynamic movement primitives,” in 2019 18th European Control Conference (ECC), 2019, pp. 2583–2588. [11] J. Aloimonos, I. Weiss, and A. Bandyopadhyay, “Active vision,” Int. J. Comp. Vis., vol. 1, pp. 333–356, 1988. [12] D. Papageorgiou, L. Koutras, and Z. Doulgeri, “A controller for reaching and unveiling a partially occluded object of interest with an eye-in-hand robot,” in 2022 IEEE-RAS 21st International Conference on Humanoid Robots (Humanoids), 2022, pp. 254–260. [13] Z. Lonˇ carevi´ c, A. Gams, S. Reberˇ sek, B. Nemec, J. ˇ Skrabar, J. Skvarˇ c, and A. Ude, “Specifying and optimizing robotic motion for visual quality inspection,” Robot. Comput.-Integr. Manuf., vol. 72, p. 102200, 2021. [14] M. A. Baumann, D. C. Dupuis, S. Leonard, E. A. Croft, and J. J. Little, “Occlusion-free path planning with a probabilistic roadmap,” in IEEE/RSJ Int. Conf. Intell. Rob. Syst. (IROS), 2008, pp. 2151–2156. [15] P. Zapotezny-Anderson and C. Lehnert, “Towards active robotic vision in agriculture: A deep learning approach to visual servoing in occluded and unstructured protected cropping environments,” vol. 52, no. 30, pp. 120– 125, 2019, 6th IFAC Conference on Sensing, Control and Automation Technologies for Agriculture AGRICONTROL 2019. [16] Y. Zaky, G. Paruthi, B. P. Tripp, and J. Bergstra, “Active perception and representation for robotic manipulation,” CoRR, vol. abs/2003.06734, 2020. [Online]. Available: https://arxiv.org/abs/2003.06734 [17] D. Papageorgiou, T. Kastritsi, Z. Doulgeri, and G. A. Rovithakis, “A passive phri controller for assisting the user in partially known tasks,” IEEE Transactions on Robotics, vol. 36, no. 3, pp. 802–815, 2020. [18] A. Vakunov, C.-L. Chang, F. Zhang, G. Sung, M. Grundmann, and V. Bazarevsky, “Mediapipe hands: On-device real-time hand tracking,” 2020. [Online]. Available: https://mixedreality.cs.cornell.edu/workshop APPENDIX A-PROOF OF LEMMA 1 Proof. Let n∈S2be a unit vector pointing towards an arbitrary direction, where S2⊂R3the set of unit vectors in R3. We can show that S(A)is enclosed by S(B), by showing that a<b, where a∈R>0: (an)⊺A−1(an) = a2n⊺A−1n= 1 and b∈R>0: (bn)⊺B−1(bn) = b2n⊺B−1n= 1, for any direction n∈S2. Notice that aand brepresent the length of the linear segments within S(A)and S(B)respectively, along the direction of n. To this aim, we have n⊺An =1 a2 and n⊺Bn =1 b2. Then n⊺An >n⊺Bn implies 1 a2>1 b2and consequently a<b.