IMPROVING HUMAN-ROBOT PHYSICAL INTERACTION COMFORT IN MATERIAL HANDLING TASKS USING A SMART PLATFORM

23 ACC JOURNAL 2023, Volume 29, Issue 1 DOI: 10.15240/tul/004/2023-1-002 IMPROVING HUMAN-ROBOT PHYSICAL INTERACTION COMFORT IN MATERIAL HANDLING TASKS USING A SMART PLATFORM Dmitry Kochubey Technical University of Liberec, Faculty of Mechanical Engineering, Institute of Mechatronics and Computer Engineering, Studentská 1402/2, 461 17, Liberec 1, Czech Republic e-mail: [email protected] Abstract The use of mobile platforms can help employees automate manual processes and streamline operations to save time and perform their tasks safely and accurately. A power-assisted vehicle to move weight around the place – solution: inexpensive, easy to apply, reliable, safe. It can adjust to various tasks, operators’ gait, loads up to 500 kg. It is a relatively inexpensive, easy-to-apply, reliable, and safe solution for moving weight. The motivation of the study is to increase efficiency and reduce physical strain on the operator in material handling tasks and to promote the implementation of this smart platform. Artificial intelligence learning methods are applied to adapt to individual operator’s experience, resulting in a personalized and more comfortable interaction with the help of Q-learning algorithm with 256 learning outcomes in adjusting controller settings: damping, mass, stiffness. Keywords Q-learning algorithms; Smart platform; AI; HRI; Material handling tasks; Human comfort criteria. Introduction With the pandemic warehouses growth in number and size [15, 16, 21], the use of mobile platforms can help employees automate manual processes and streamline operations to save time and perform their tasks safely and accurately. There are many types of wheeled devices that we encounter in people’s daily lives. Industrial carts and transporters are widely used in various industries and warehouses, while hospitals use bed movers and wheelchairs to transport patients. Supermarket trolleys make shopping easier, and baby strollers allow us to take our children on long walks. Multiple earlier studies [1, 5, 6] demonstrated that the use of manual vehicles can increase human efficiency and reduce stress in manual handling tasks. Many groups of people, including professional workers, parents, disabled individuals, and customers, use some form of mobile vehicle to solve a material handling task at some point in their lives. Material handling can expose workers to risk factors for low-back disorders, such as lifting, bending, twisting, pulling, pushing, and maintaining static postures. Pushing and pulling activities make up almost half of all manual materials handling [10]. This study focuses specifically on carrying activities using industrial power-assisted carts, which are typically manipulated by pulling backwards and pushing forwards with two hands. Pushing is generally preferred to pulling because it is safer. When pulling, the operator’s feet may be run over by the cart, especially if it is powered, and the arm is stretched behind the body in an awkward position that increases the risk of injury. Pulling while walking backwards is also dangerous because the operator cannot see the path of travel. Research has shown that people can usually 24 exert higher push forces than pull forces [13]. While pulling may be necessary in some situations, it should be avoided whenever possible and minimized when necessary. 1 Research Objectives The goal of the current research is to address the issue of discomfort experienced during interactions between humans and industrial carts. The study is focused on situations where an operator uses a cart to navigate through a space that may contain obstacles and targets (such as a supermarket, warehouse, hospital, or transportation hub). The operator may choose targets and paths that are not always optimal, and their physical and mental state, along with the weight of the load, may change over time. It is aimed to identify criteria that reflect the operator’s comfort and satisfaction, and use this information to adjust a support system to improve comfort. To do this, a chain of experiments is designed and conducted involving linear and curvilinear cart movement, using impedance controllers to control both linear and rotational motion in the latter case. The effects of various controller settings on operator comfort are examined and a system that allows adjusting these settings automatically using an optimization method is developed. The article is organized as follows. The next section reviews the existing literature on the topic, highlighting relevant studies and theories that provide a foundation for the current research. The research design, Q-learning algorithm and human comfort criteria that were used in the study are discussed in methods. The following section presents the findings of the study. The results and their possible practical implications as well as limitations encountered in the current study are listed and explored in the discussion and limitations section. The article concludes with the summary of the work and suggests future directions for research in the field. 2 Theoretical Background Mechanical impedance control is a control method that involves the manipulation of the mechanical impedance of a system to achieve precise and robust control of its motion. The concept of mechanical impedance control was first introduced by Hogan from the Massachusetts Institute of Technology (MIT) [7, 8]. Mechanical impedance control has had a significant impact on the field of robotics and control engineering. It provides a flexible and robust approach to controlling the motion of robotic systems and other mechanical systems that interact with the environment. By modulating the mechanical impedance of the system, it is possible to achieve a desired motion or interaction with the environment, while also allowing the system to adapt to changes in the load or the environment. Mechanical impedance control has been widely applied in the control of robots and manipulators (see, e.g. [14]), as well as in the development of assistive devices, such as exoskeletons and powered prostheses [3]). It was also used in the development of haptic interfaces, which provide touch feedback in virtual environments, and in the control of rehabilitation devices, such as robotic arm and leg trainers. There are a few examples of a power-assisted control system based on a compliance controller for exoskeletons, mobile platforms and wheelchairs. The first one was developed at the Rehabilitation Institute of Chicago [2]. The second powered exoskeleton was developed by researchers at the National Institute of Advanced Industrial Science and Technology (AIST) in Japan [20]. These exoskeletons are designed to assist people with lower limb paralysis to walk by providing powered assistance to their legs. The exoskeletons use a compliance controller to adjust the impedance of the powered joints in response to the user’s movements and the environment. It allows the exoskeleton to adapt to changes in the user's motion and the terrain, providing a more natural and comfortable walking experience. 25 Two examples of a power-assisted control system based on a compliance controller for wheelchairs include the HapticMaster developed by researchers at the Netherlands [22] and the Kinetisense developed by researchers at the University of California [18]. Both represent a powered wheelchair that uses a compliance controller to adjust the impedance of the wheels in response to the user's movements and the environment. The compliance controller allows the wheelchairs to adapt to changes in the user’s motion and the terrain, providing a more natural and comfortable ride for the operator. An examples of a power-assisted control system used for mobile platforms and trolleys could be AGV (Automated Guided Vehicle) developed by researchers at the University of Hong Kong [12]. The compliance controller adjusts the impedance of the platform’s motion in response to the load and the environment, enabling it to navigate around obstacles and maintain a stable and comfortable ride for the operator. These types of power-assisted control systems are described in Table 1. All of these projects have the potential to greatly improve the mobility and independence of people with mobility impairments, particularly in challenging environments such as rough terrain or steep slopes and the efficiency and safety of logistics operations. Tab. 1: Comparison of the types of power-assisted control systems Project name Year Research team members Main concept Kinetisense 2013 Hargrove, Peshkin Powered wheelchair with compliance controller for wheels HapticMaster 2011 Henze, Wahl, Buss, Kohl, Müller, Hertzberg Powered wheelchair with compliance controller for wheels Powered exoskeleton 2010 Aoyama, Iwamoto, Nakano Powered exoskeleton with compliance controller for legs AGV 2004 Wong, Liu, Leung Mobile platform with compliance controller for logistics operations Impedancecontrolled exoskeleton 2004 Reinkensmeyer, Herr Powered exoskeleton with compliance controller for legs Source: Own Manipulation with the object of interest requires a physical interaction. In order to fulfil the task requirements, the user chooses desired impedance that can be expressed by equation (1): Md(𝑥󰇘 − 𝑥󰇘𝑑) + 𝐵d(𝑥󰇗 − 𝑥󰇗𝑑) + K𝑑(𝑥 − 𝑥𝑑) = −fe, (1) where M𝑑,B𝑑 and K𝑑 are positive constants that represent the desired inertia, damping and stiffness, respectively. From the equation (1) we can find the acceleration reference (2): 𝑥󰇘𝑟= 𝑥󰇘𝑑+ 𝑀𝑑 −1 ∙[−fe+ 𝐵d(𝑥󰇗𝑑− 𝑥󰇗)+ K𝑑(𝑥𝑑− 𝑥)] (2) For admittance control, the control force is a position-controller designed to track the trajectory 𝑥 = 𝑥𝑑. Trajectory tracking is implemented using a PD controller with positive gains K𝑝 and K𝑑 (3): fr= K𝑝(𝑥𝑑− 𝑥) + K𝑑𝑥󰇗 (3) The simplified impedance controller could be written in the form (4): 26 Md(𝑥󰇘 − 𝑥󰇘𝑑) + 𝐵d(𝑥󰇗 − 𝑥󰇗𝑑) = −fe, (4) where Md is mass, Bd refers to damping, and external force is −fe. The values of Md and Bd depend on the physical properties of the system, and the desired values 𝑥󰇘𝑑 and 𝑥󰇗𝑑 can be used to specify the desired behavior of the system. Some researchers [4] found that the spring component of an impedance controller does not significantly affect the interaction process. 3 Methods In robotics, the control of human-industrial cart interaction can be performed using a variety of techniques, depending on the specific application and requirements. In this set of experiments, Human-Robot collaboration approach was used. The platform could interact with the human, for example, by guiding the load to the desired location, assist the human with turns and stops as well as adjust to his driving technique using adaptive control. In its turn, adaptive control in human-robot collaboration refers to the ability of a robot to adjust its behavior in response to changes in the human operator's behavior, skill level, or preferences. This allows the robot to adapt to the human’s abilities and work style, leading to a more efficient and natural collaboration. Adaptive control in human-robot collaboration is currently a multidisciplinary field that combines concepts from control systems, robotics, human-computer interaction, and cognitive science. In this part of the article, the control algorithm developed for robust and safe human robot interaction is described. An essential component for the solution is the impedance controller [9], which could be used to represent and evaluate human-operator dynamics and to control supporting effort of the mobile platform side during the interaction process. This study aims to improve human-robot physical interaction comfort in material handling tasks by incorporating AI learning methods into a smart platform. As the first step, the human operator's experience and estimation are utilized to adapt to the impedance of the platform, leading to a more personalized and comfortable interaction. This is achieved by incorporating the two strategies demonstrated in previous studies on force field tasks, as described in numerous references [11, 17]. According to these studies, humans adjust their impedance in response to perturbations by applying mainly two strategies: 1) increasing impedance through co-interaction in the case of unpredictable perturbations, 2) learning a feed-forward command to offset predictable perturbations. By incorporating these strategies into the smart platform, we hope to minimize position error and energy consumption during material handling tasks, leading to improved efficiency and reduced physical strain of the operator. On the other hand, the smart platform adjusts its own interaction strategy by changing the impedance parameters according to the correlation between detected features and human feelings, such as mass and damping coefficient for the former and level of operator’s satisfaction for the latter. This enables the platform to adapt to the operator’s individual experience in a timely manner and ensure interaction that is more comfortable. The adjustment process is accomplished using a Markov Decision Process (MDP). An MDP is a mathematical framework that is widely used to model decision-making situations. In the context of improving human-robot physical interaction comfort in material handling tasks, the MDP is used to model the decision-making process of the smart platform in adjusting its impedance parameters. An MDP is defined by the following components: 27  A set of states of the world (S), which represent the various conditions or situations that the platform may encounter during material handling tasks.  A set of actions (A), which represent the different strategies that the platform can adopt to respond to the detected states.  A transition function (T(s, a, s’)), which describes the probability of moving from one state to another state after taking an action.  A reward function (R(s, a)), which assigns a reward or cost to each state-action pair.  A policy (π), which defines the strategy the platform uses to select actions in each state, based on the rewards and the transition probabilities. The platform uses this information to determine the optimal impedance adjustment that maximizes the expected reward over time, leading to improved comfort and efficiency in material handling tasks. The policy is updated iteratively as the platform learns from experience and improves its decision-making process over time. A reinforcement learning algorithm was implemented to include human feelings in the control system. As defined in Sutton and Barto’s book ‘Reinforcement Learning: An Introduction’ [19], “Reinforcement learning is an approach to learning from interaction with an environment, by trial and error, and receiving rewards or penalties for different actions.” [19, p. xi] Reinforcement learning algorithms are built on the concepts of MDP. The first task when designing Q-Learning system is to define the environment. The environment consists of states, actions and rewards. The agent uses states and rewards as inputs and generates his actions as outputs. The Q-Learning algorithm was first introduced in the framework of the PhD thesis of Watkins in 1989 [24] and developed later in 1992 [23], which stands for a model-free reinforcement learning algorithm that uses the concept of the action-value function, also known as the Q-function. The Q-function is an estimate of the expected longterm reward for a given state-action pair, and is updated iteratively as the agent interacts with the environment. 3.1 States The number of states that may occur is limited and finite. Each possible setting of the impedance controllers can be considered as a state, and the agent can only be in one state at a time. This means that only one set of impedance controller settings can be selected and evaluated in each step. The study utilized four parameters for each impedance controller coefficient, which resulted in a total of 256 possible combinations. It is believed that this number of combinations is sufficient to demonstrate the learning process of the platform, although the trade-off between the flexibility of the settings and the time required for the learning process must be considered when determining the number of coefficients to be used in future studies. 3.2 Actions The number of possible actions is finite. The agent will always need to choose from a fixed number of possible actions as was proved by the results of the regression analysis that was carried out for damping and mass coefficients. A set of possible actions was defined in the following way: the agent could apply two actions (increase or decrease) per each of the four parameters and carry out an additional “do nothing action” when no change was required. The change of inertial and dumping components of the impedance controller leads to a change in the cart dynamics. 28 3.3 Rewards The reward system works as follows. The agent checks if the interaction dynamics is positive by comparing values of mean values and standard deviation for the current step and the previous step. Additionally, the agent checks if there is no emergency situation by analyzing the E-stop button state. Peaks of the interaction force have to be avoided as well. If a human operator thinks that the current settings are convenient for him he might give positive feedback. In the end, the rewards for different criteria are totaled. If none of the criteria were met, the reward is set to a negative one. 4 Results Q-learning algorithm was implemented inside the high-level controller that is Raspberry Pi 4 in our case by using Python. The information about process values (interaction forces, odometry) is supplied to high-level controller from low-level controller using the serial port. Using the same link information about the actual impedance controller parameters provided to low-level controller. Protocol uses a CRC data check. The data of biological markers is read from a smart band using a BLE protocol. The console output of the learning process is shown in Figure 1. Source: Own Fig. 1: Console output of the learning process Information consists of the current episode number, the number of the step inside the episode, the selected action, obtained reward and the new set of impedance controller parameters to be tested. The diagram of the Q-Learning process can be presented in the shape of a pseudo-code shown in Figure 2. In Figure 3 the graphic visualization of the Q-table values is illustrated. The representation of the high-rating areas is depicted in yellow, while the low-rating areas are shown in blue. At the start of the interaction process, the values in the Q-Table are equal. However, as soon as the algorithm takes action, the system state will change, and the corresponding value in the QTable will be updated based on the reward information. The quality and speed of the reinforcement learning process are influenced by the teacher. If the human operator utilizes the user button to provide positive feedback or the e-stop to give negative feedback, it can significantly accelerate the learning process. 29 Source: Own Fig. 2: Learning function of the Q-Learning algorithm presented in pseudo-code Source: Own Fig. 3: Dynamic change of the Q-value during the learning process The changes in the Q-Table over time can be observed by examining the Q-Learning dynamics. The Q-Table is visualized using a color map (heat map). Initially, the Q-Values are relatively similar, and a significant portion of the color map is depicted in yellow. Over time, the color map becomes darker as the algorithm receives negative feedback about the impedance controller settings. In the long run, the majority of the color map is covered in dark blue and green, indicating the negative impact of the impedance controller setting on the interaction process. Only a tiny yellow line remains, representing the impedance controller settings that are fully responsive to the human operator’s intention. The impedance controller settings that correspond to the maximum value of the Q-Table can be obtained by selecting the corresponding state. By observing and learning from the operator’s behavior, the mobile platform can adapt its parameters to match the operator, which can improve its performance in the presence of disturbances and its ability to recover from errors. The mobile platform can learn to anticipate and avoid dangerous situations, such as collisions with obstacles or other vehicles. Several scientific criteria are used to determine the robustness and safety of a mobile platform. These criteria include performance in the presence of disturbances, recovery from errors, safety, passivity, and efficiency. These criteria have been developed and studied over time by many engineers and scientists and have been formalized in various works, research, and standards. For instance, IEEE and ISO have developed standards for the safety and 30 performance of mobile robots and Automated Guided Vehicles (AGVs). The specific criteria and standards used to evaluate the robustness and safety of a mobile platform will depend on the specific application and environment in which it will be used. 5 Limitations and Discussion Having specified these criteria, we believe that the current mobile platform might be robust and safe due to the following list of reasons: the maximum speed is restricted to prevent dangerous situations and unexpected behavior. An emergency stop button (E-stop) has been implemented to ensure additional safety. Moreover, the main power switch is available to control the power supply to the system. To further prevent dangerous situations, an interlock has been implemented to avoid instant changes in direction at high-velocity set-points. The system is also designed to slow down, but not accelerate in the opposite direction without reaching a low speed to prevent sudden changes in direction. The control system with the impedance controllers of rotational and translational motion was implemented in the experimental platform. It allows supporting human operator during the linear drive and turns. By analyzing the interaction characteristics, it was possible to obtain the dynamics relevant to the material handling task and to identify the physical measures, emotional feedback and biological markers that were used as additional sources of information to improve interaction. The platform works well with loads up to 500kg. Currently, the learning outcomes include a matrix of 256 components, which is not a limit for the algorithm; however, it is believed to be sufficient for demonstration purposes. Conclusion A mathematical and experimental model of the industrial power-assisted cart was developed. A great amount of work was performed in powered mobile platform programming and control system implementation. Therefore, artificial intelligence (AI) methods were employed to adjust the controller settings so that an operator can manipulate an industrial cart loaded up to 500 kg with minimum physical effort and ultimate comfort. Human estimation criteria that characterize the satisfaction and comfort from the human-powered cart interaction process were synthesized. Based on these synthesized criteria, the human-powered cart interaction control algorithm was developed using AI methods (Q-learning). The performance of the proposed solution for the developed industrial cart was tested and verified. The research work contributed the following theoretical input into the field of technical cybernetics – the use of Q-learning algorithm in adjusting controller settings so that the mobile platform could successfully and effectively adapt to the unique gait and tasks of any operator it assists. Literature [1] BANERJEE, S.; ACHARYA, K. N.; CHATTOPADHYAY, D. P.: Studies on energy expenditure of rickshaw pullers. [online]. Indian Journal of Physiology and Pharmacology. 1959, Vol. 3, Issue 3, pp. 147–160. Available from WWW: https://ijpp.com/IJPP%20archives/1959_3_3/1959_3_3_toc.php [2] BLAYA, J. A.; HERR, H.: Adaptive Control of a Variable-Impedance Ankle-Foot Orthosis to Assist Drop-Foot Gait. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2004, Vol. 12, Issue 1, pp. 24–31. DOI: 10.1109/TNSRE.2003.823266 [3] BAUD, R.; MANZOORI, A. R.; IJSPEERT, A.; BOURI, M.: Review of control strategies for lower-limb exoskeletons to assist gait. Journal of NeuroEngineering and Rehabilitation. 2021, Vol. 18. DOI: 10.1186/s12984-021-00906-3 31 [4] DUCHAINE, V.; GOSSELIN, C. M.: General Model of Human-Robot Cooperation Using a Novel Velocity Based Variable Impedance Control. In: Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (WHC’07). Tsukuba, Japan, IEEE, 2007, pp. 446–451. DOI: 10.1109/WHC.2007.59 [5] HAISMAN, M. F.; WINSMANN, F. R.; GOLDMAN, R. F.: Energy cost of pushing loaded handcarts. Journal of Applied Physiology. 1972, Vol. 33, Issue 2, pp. 181–183. DOI: 10.1152/jappl.1972.33.2.181 [6] JÄGER, M.; LUTTMANN, A.; LAURIG, W.: The load on the spine during the transport of dustbins. Applied Ergonomics. 1984, Vol. 15, Issue 2, pp. 91–98. DOI: 10.1016/0003-6870(84)90278-3 [7] HOGAN, N.: Impedance control of industrial robots. Robotics and ComputerIntegrated Manufacturing. 1984, Vol. 1, Issue 1, pp. 97–113. DOI: 10.1016/07365845(84)90084-x [8] VOLPE, B. T.; KREBS, H. I.; HOGAN, N.; EDELSTEIN, L.; DIELS, C.; AISEN, M.: A novel approach to stroke rehabilitation: Robot-aided sensorimotor stimulation. Neurology. 2000, Vol. 54, Issue 10, pp. 1938–1944. DOI: 10.1212/WNL.54.10.1938 [9] KANA, S.; TEE, K.-P.; CAMPOLO, D.: Human–Robot co-manipulation during surface tooling: A general framework based on impedance control, haptic rendering and discrete geometry. Robotics and Computer-Integrated Manufacturing. 2021, Vol. 67. DOI: 10.1016/j.rcim.2020.102033 [10] KUMAR, S.; NARAYAN, Y.; BACCHUS, Ch.: Symmetric and Asymmetric TwoHanded Pull-Push Strength of Young Adults. Human Factors. 1995, Vol. 37, Issue 4, pp. 854–865. DOI: 10.1518/001872095778995526 [11] LACHNER, J.; SCHETTINO, V.; ALLMENDINGER, F.; FIORE, M. D.; FICUCIELLO, F.; SICILIANO, B.; STRAMIGIOLI, S.: The influence of coordinates in robotic manipulability analysis. Mechanism and Machine Theory. 2020, Vol. 146. DOI: 10.1016/j.mechmachtheory.2019.103722 [12] LEE, Ch. W.; WONG, W. P.; IGNATIUS, J.; RAHMAN, A.; TSENG, M.-L.: Winner determination problem in multiple automated guided vehicle considering cost and flexibility. Computers & Industrial Engineering. 2020, Vol. 142. DOI: 10.1016/j.cie.2020.106337 [13] LEE, K. S.; CHAFFIN, D. B.; HERRIN, G. D.; WAIKAR, A. M.: Effect of handle height on lower-back loading in cart pushing and pulling. Applied Ergonomics. 1991, Vol. 22, Issue 2, pp. 117–123. DOI: 10.1016/0003-6870(91)90310-e [14] LEE, S.-D.; JUNG, S.: Human-actuator collaborative control by a novel frequencydivision technique for linear maneuverability of control moment gyroscopic actuators. Mechatronics. 2018, Vol. 55, pp. 224–233. DOI: 10.1016/j.mechatronics.2018.05.001 [15] NAR: Real Estate in a Digital Age 2018 Report. [online]. National Association of REALTORS®, 2018. [accessed 2023-02-01]. Available from WWW: https://www.nar.realtor/sites/default/files/documents/2018-real-estate-in-a-digitalworld-12-12-2018.pdf [16] NREI: National Real Estate Investor®: 2019 Outlook. [online]. 2019. [accessed 202302-02]. Available from WWW: https://mf.freddiemac.com/docs/nrei.pdf