Human Motion Prediction using Interacting Multiple Model Filtering enhanced by a Kinematic Model

Human Motion Prediction using Interacting Multiple Model Filtering enhanced by a Kinematic Model Michele Ferrari∗† and Manuel Beschi∗† ∗Dipartimento di Ingegneria Meccanica e Industriale, University of Brescia, Italy †Institute of Intelligent Industrial Technologies and Systems for Advanced Manufacturing, National Research Council of Italy Corresponding author: Michele Ferrari ([email protected]) Abstract—Predicting human motion enhances safety and efficiency in human–robot collaboration. We evaluate a predictor based on an Interacting Multiple Model (IMM) estimator combined with a human dynamics and control model. This approach is compared against a baseline Unscented Kalman Filter (UKF) using a constant acceleration model (CA). Additionally, we introduce a human kinematic model to predict joint angles instead of keypoints, enforcing kinematic constraints. Index Terms—Human-motion prediction, Safe human-robot interaction, Unscented Kalman Filter I. INTRODUCTION In human–robot collaboration, predicting human motion enables the design of safe robot trajectories (per ISO/TS 15066:2016 [1]), allowing path and task adjustments to prevent collisions, avoid unnecessary stops, and anticipate human actions. Prior methods for human pose prediction include Recurrent Neural Networks (RNNs) [2], Inverse Optimal Control [3], graphical models such as Hidden Markov Models (HMMs) [4], and other learning-based techniques [5]. We propose instead a pose prediction method based on an Interacting Multiple Model (IMM) estimator [6] and a simplified human motion model suited for scenarios with minimal tracking data. II. METHOD A. Problem definition The human position is defined as the set of nbody key points pi∈R3in Cartesian space: y∈R3n= pT 1, pT 2, . . . , pT nT. These key points can be readily obtained using vision-based skeleton tracking systems. Following [7], we introduce a human dynamic model that describes body motion: (y(t) = hh(xh(t)) ˙xh(t) = fh(xh(t), u(t)) (1) where xhis the dynamic state, uis the input signal, fh(·)is the dynamic function, and hh(·)is the output function. We also define a human control model that computes the control signal u(t)generated by the human and used by the dynamic model in (1): (u(t) = hc(xh(t), xc(t), r(t)) ˙xc(t) = fc(xh(t), xc(t), r(t)) (2) where xcis the controller internal state, ris an exogenous signal (e.g., the target object position during grasping), fc(·) Fig. 1: The skeletonized operator in the collaborative robotic cell. is the controller dynamic function, and hc(·)is the controller output function. The full dynamic model is obtained by combining and discretizing equations (1) and (2). The state x(k|k)and covariance P(k|k)are estimated by a state observer. In the predict step, it computes x(k+ 1|k)and P(k+ 1|k)at the time k+ 1 based on information available at time k. In the update step, it corrects the estimate with the actual measurements computing x(k|k)and P(k|k). In this work, the predict step is iterated ntimes to compute the nstep-ahead prediction of the future position y(k+n|k). B. Predictor implementation The most straightforward approach operates directly in the Cartesian space of human keypoints. In this case, the state vector is defined as xh(t)=[y(t)T, v(t)T]T, where v(t) = ˙y(t)∈R3nrepresents the human velocity. The dynamic system in (1) can then be modeled as a double integrator between acceleration uand position y. A more sophisticated strategy defines the state vector as xh= [qi], a set of joint variables qifor i∈[1, . . . , nDoF ] that represent the human’s kinematic configuration. The statetransition model fhremains a double integrator, now applied to joint acceleration and position. Unlike the previous approach, the output function hhinvolves computing the forward kinematics of the human model: y(t) = fkin(xh(t)) (3) where the fkin(·)is the nonlinear forward kinematics. In this work, we implement a 28-DoF kinematic model1(Fig. 2). The human control model in (2) can be complex and nonlinear when exogenous information is available. However, when only pose tracking data is provided, it is possible to 1https://github.com/JRL-CARI-CNR-UNIBS/human kinematic model.git 2025 I-RIM Conference October 17-19, Rome, Italy ISBN: 9788894580570 10.5281/zenodo.17629762 133 TABLE I: Performance Metrics computed on the Validation Set Prediction Window Model REACH-TO-GRASP WALKING PASSING-BY µe(×10−4m) σe(×10−2m) µe(×10−4m) σe(×10−2m) µe(×10−4m) σe(×10−2m) 0.1 s(one step) CA −1.79 1.06 −32.4 1.36 −55.4 1.64 IMM −1.46 0.78 −31.5 1.07 −54.5 1.38 0.3 s(three steps) CA 0.530 3.25 −35.4 4.03 −112 7.33 IMM −2.37 2.64 −39.5 3.93 101 6.78 0.5 s(five steps) CA 1.81 6.42 −40.2 7.86 −171 11.1 IMM −2.23 4.93 −43.3 6.72 −148 10.0 Fig. 2: The reference frames of the human kinematic model. assume that the acceleration uis a piecewise constant function. In this case, (2) simplifies to u(t) = 0 when the velocity is constant, or (u(t) = xc ˙xc= 0 (4) when the acceleration is constant. The Interacting Multiple Model (IMM) algorithm efficiently combines Unscented Kalman Filters based on constant acceleration (CA) and constant velocity (CV) control models. It handles transitions between these modes by updating their associated probabilities through a Markov chain. III. SETUP AND EXPERIMENTS Experiments were conducted in an industrial collaborative robotic cell using a StereoLabs ZED RGB-D camera. The camera runs proprietary skeletonization software at ∼25 Hz to detect human key points. Ten subjects performed various tasks (e.g., reach-to-grasp, walking, passing-by) at different speeds. The study evaluated the predictive accuracy and uncertainty bounds of the proposed model under varying conditions. IV. RESULTS AND DISCUSSION Let e(k) = y(k|k)−y(k|k−n)denote the keypoint-wise error between the current filter estimate and the n-step-ahead prediction. The following error metrics were evaluated: •the mean value µe=E(e)of the key-point-wise error. This metric verifies the assumption of estimator meanunbiasedness. A lower µeindicates a less biased n-stepahead prediction. •the standard deviation σe=qE(e−µe) (e−µe)T of the key-point-wise error. This value represents the spread of the prediction error around its mean value and thus assesses the accuracy of the n-step estimate. Table Ireports the error metrics for each condition, aggregated by task-specific key points. Task velocity aggregation was also performed to derive a comprehensive global metric. Overall, the mean error (µe) remains consistently low across tasks, indicating minimal bias in the predicted estimates. Although both the Constant Acceleration (CA) and Interacting Multiple Model (IMM) filters yield similar values for µe, the IMM shows a smaller standard deviation (σe), leading to reduced Root-Mean-Squared-Error (RMSE) across all tasks and prediction horizons. V. CONCLUSIONS AND FUTURE WORKS This study presents a human motion prediction framework based on sequential open-loop UKF prediction steps using a simple dynamic and control model, achieving reliable forecasts over a 0.5-second horizon. The Interacting Multiple Model (IMM) estimator effectively manages transitions between motion modalities without relying on complex, learned systems. Future work will aim to improve prediction accuracy by increasing the complexity of the human control model, potentially by learning typical control dynamics from data. REFERENCES [1] International Organization for Standardization, “ISO/TS 15066:2016 Robots and robotic devices — Collaborative robots.” 2016. [2] W. Liu, X. Liang, and M. Zheng, “Dynamic model informed human motion prediction based on unscented kalman filter,” IEEE/ASME Transactions on Mechatronics, vol. 27, no. 6, pp. 5287–5295, 2022. [3] J. Mainprice, R. Hayne, and D. Berenson, “Goal set inverse optimal control and iterative replanning for predicting human reaching motions in shared workspaces,” IEEE Transactions on Robotics, vol. 32, no. 4, pp. 897–908, 2016. [4] H. S. Koppula, R. Gupta, and A. Saxena, “Learning human activities and object affordances from rgb-d videos,” The International Journal of Robotics Research, vol. 32, no. 8, pp. 951–970, 2013. [5] D. Fridovich-Keil, A. Bajcsy, J. F. Fisac, S. L. Herbert, S. Wang, A. D. Dragan, and C. J. Tomlin, “Confidence-aware motion prediction for real-time collision avoidance,” The International Journal of Robotics Research, vol. 39, no. 2-3, pp. 250–265, 2020. [6] H. Blom and Y. Bar-Shalom, “The interacting multiple model algorithm for systems with markovian switching coefficients,” IEEE Transactions on Automatic Control, vol. 33, no. 8, pp. 780–783, 1988. [7] M. Ferrari, S. Sandrini, C. Tonola, E. Villagrossi, and M. Beschi, “Predicting human motion using the unscented kalman filter for safe and efficient human-robot collaboration,” in 2024 IEEE 29th International Conference on Emerging Technologies and Factory Automation (ETFA), 2024, pp. 1–8. 134