Software Architecture and Human-Centric Design Methodology in Human-Robot Collaborative Production Systems

Human Aspects of Advanced Manufacturing, Production Management and Process Control, Vol. 146, 2024, 195–204 https://doi.org/10.54941/ahfe1005165 Software Architecture and Human-Centric Design Methodology in Human-Robot Collaborative Production Systems Spyridon Koukas1, Raimund Broechler1, Kevin Haninger2, Milan Gautam3, Fernando Ubis3, Urko Esnaola Campos4, Iveta Eimontaite5, Sarah Fletcher5, Nikos Dimitropoulos6, and Sotiris Makris6 1Netcompany-Intrasoft, 2B Rue Nicolas Bové, Luxembourg City 1253, Luxembourg 2Department of Automation, Fraunhofer Institute for Production Systems and Design Technology, Pascalstraße 8-9, Berlin 10587, Federal Republic of Germany 3Visual Components, Vänrikinkuja 2, Espoo 02600, Finland 4Fundacion Tecnalia Research & Innovation, Parque Cientifico Y Tecnologico De Gipuzkoa Paseo Mikeletegi 2, Donostia / San Sebastian 20009, Spain 5School of Aerospace, Transport and Manufacturing, Cranfield University, Cranfield, UK 6Laboratory for Manufacturing Systems & Automation, Department of Mechanical Engineering and Aeronautics, University of Patras, Patras 26504, Greece ABSTRACT Over the last years both Research and Industry have allocated significant effort to address the requirement for flexible production by introducing technologies that allow humans and robots to coexist and share production tasks safely. The human-robot coexistence and collaboration in cageless environments introduces challenges that are both of technical and social nature. A human robot collaborative system can be viewed as a sociotechnical work system that can both increase production KPIs and improve the quality of life for people working in HRC environment. In addition, effective HRC requires acceptance and trust from the side of the operators and clear communication between operators and robots. This work presents a software architecture approach that enables personalized and human-centric HRC and aims to improve ergonomics, cognitive factors and acceptance. In this context, key technology solutions for different HRC aspects such as training, design and production are also presented. Keywords: Robotics, Architecture, Human-centric design, Ergonomics, Human-robot collaboration INTRODUCTION The increasing interest in flexible manufacturing and HRC during the last years, by both Research and Industry, should not come as a surprise. The increased use of Information and Communication Technologies (ICT) and robots in workplaces has been changing the world of work. Between 2000 © 2024. Published by AHFE Open Access. All rights reserved. 195 196 Koukas et al. and 2019, the real value of ICT capital per worker in Europe rose by 91%. The robot exposure, measured by the number of industrial robots per 1,000 workers, increased by 140% (Albinowski and Lewandowski, 2024). The introduction of robots in production offers a number of advantages, such as Quality improvement, Workers’ health by undertaking unhealthy manufacturing operations and Reduction of throughput time (Lien, 2014). The HRC approach and flexible manufacturing offer a lot of advantages as they can combine traits such as robot repeatability and strength with human dexterity. Modern markets require a short development cycle yielding low-cost, high-quality goods in sufficient quantity to meet demand. This makes flexibility an increasingly important attribute to manufacturing (Chryssolouris, 2006). In this regard, cooperating robots offer a radically new paradigm for uplifting the capability of manufacturing systems to deal with the modern manufacturing flexibility requirements (Makris, 2020). Decisions regarding the design and operation of manufacturing systems require technical understanding and expertise, as well as the ability to satisfy certain business objectives. Thus, a combination of engineering and management disciplines are required to provide a decision-making framework for manufacturing (Chryssolouris, 2006). The introduction of new technologies, such as human-robot collaboration (HRC) applications in a manufacturing system needs also to address some prominent challenges and tensions inherent in technology adoption; job displacement, employee’s acceptance, trust, and privacy (Leesakul et al., 2022). This work aims to give an overview of an HRC system from the perspective ongoing CONVERGING project. In particular, CONVERGING envisions a HRC production system where production resources actively take actions to improve the experience of their human collaborators in multiple aspects. In this sense, the HRC production is examined as a sociotechnical production system where technical and social scientific expertise contributes to the improvement of human experience. A HUMAN CENTRIC ARCHITECTURE APPROACH FOR HRC The CONVERGING software architecture aims to integrate and connect smart and reconfigurable production systems including multiple autonomous resources and human operators in diverse production environments. The architecture supports the human-centric vision of the project with specific workflows and software modules that are dedicated to the improvement of the user experience. The system is centrally orchestrated from the AI Station Controller (AI-SC) module. The AI-SC module can accept production requests from an external system and also offers a UI that allows the user to control and monitor the HRC production. The first step of the AI-SC after a production request has been received is to convert production request to a set of required tasks that need to be executed and then request the production plan from the Dynamic Work Reorganization (DRW) module. The request of the AI-SC includes the current work assignments, the location and availability of each resource and the required tasks that need to be completed. Software Architecture and Human-Centric Design Methodology 197 Figure 1: Converging human centric architecture approach. The DWR produces an optimized sequence of tasks and assignments of tasks to resources. In order to do so, the DWR can access the results of past work plans and also request simulation of specific tasks from the AI enabled Digital Twin module. This planning phase already provides a highlevel improvement of the human operators’ experience, as the safety and ergonomy of the overall production is one of its configurable optimization targets. The assignment of tasks with bad ergonomy to robots, is a first step of improving the ergonomy in a high level. An approach he novelty of the proposed framework relies on the fact that the implemented criteria are focused on improving, among others, the safety and ergonomics of Human–Robot Collaborative workplaces. Once the AI-SC receives the sequence and assignments of tasks, it converts them to a production schedule that is suitable for execution. Each task is converted to a set of actions that depends on the task itself and the assigned resource. Then the AI-SC starts the execution of the production schedule by requesting and monitoring the execution of specific actions from the different modules.The execution of collaborative tasks is delegated to the Collaborative Robot Control (CRC) that is responsible for controlling robotic resources in a collaborative environment. This is achieved in collaboration with the Autonomous Robot Behavior Adjustment (ARBA) module and the User Experience and Ergonomics (UEAE) module. The ARBA module utilizes models of the production tasks including the robots’ actions and evaluates the task performance over data that are collected during execution to improve and adapt the robot control policy. The UEAE module is responsible to optimize the robot’s pose and motion characteristics in order to optimize ergonomic and cognitive factors of the human operator. Prior each task execution the AI-SC supplies the CRC with the execution context, including the pending task and related parameters and information of the operator that participates in the collaborative task.In addition, the CRC module receives real-time data from the Perception and Autonomy module (PAM), such as 3D data representation of the environment, force readings from force sensors and the operator’s position and stance. This feedback allows the CRC to 198 Koukas et al. control the robot in real-time, in a way that improves the operator’s position and ergonomy. Furthermore, a Safety Assessment and Monitoring system is always connected to the robot, ensuring the safety of the human operator. Ergonomy During Task Design The integration of ergonomic principles in manufacturing processes is essential for optimizing human-machine interactions, minimizing physical strain, and reducing the risk of musculoskeletal disorders (MSDs) among workers. By designing new processes with consideration for human factors, such as posture, reach, and visibility, employee comfort, productivity, and job satisfaction can be enhanced (Karwowski and Marras, 2003). The same should be applied when introducing human-robot collaboration as this type of automation not only can take over the repetitive and strenuous tasks, but adjusts physical collaboration aspects to ensure the human workers posture is the most optimal for the given task. Moreover, a well-implemented ergonomic approach leads to decreased absenteeism, lower rates of workplace injuries, and improved product quality (Chaffin and Andersson, 1991). Using appropriate software tool different tasks can be modelled and evaluated for ergonomic risk, this evaluation allows the design of tasks with improved ergonomy. Visual Components 4.0 (VC4.0) simulation software is based on discrete event simulation for production and manufacturing environments. It includes human model as digital component with modifiable properties depicted in Figure 1. Figure 2: Human model in VC4.0 with modifiable properties. Software Architecture and Human-Centric Design Methodology 199 The human model in VC4.0 models a realistic human skeleton model commonly used in ergonomics science illustrated in Figure 1. The digital human model gives access to control different joints of digital human shown in Figure 2. This allows designers of manufacturing facilities to program by themselves and simulate the human model and check ergonomics issues in the designed system even before the establishment of the production cell or manufacturing line. If there are existing ergonomics challenges observed in established facility, using pose tracking sensors and mapping sensor data to the digital model data, designers can evaluate the ergonomics risk, improved the work setup in digital platform and test the improvements in ergonomics digitally and make changes in the real environment. The process also ensures that the data is preserved in the digital model for future use. The data generated out of digital human model can be used for Rapid Upper Limb Assessment (RULA) which is used in ergonomics to evaluate the exposure of individual workers to ergonomic risk factors associated with upper extremity. Joint angles from digital human model can be used for this purpose like the approach (Plantard et al., 2017) where joint angles obtained from Kinect sensor was used for RULA. Figure 3: a) Skeleton model from Plantard et al., 2016, (HC) hip center, (SP) spine, (SC) shoulder center, (H) head, (SL) left and (SR) right shoulders, (EL) left and (ER) right elbows, (WL) left and (WR) right wrists, (HL) left and (HR) right hips. b) Kinematic model of human in visual components 4.0. 200 Koukas et al. Figure 4: Joints control for digital human model in VC4.0. TECHNOLOGY ENABLERS XR Environments The purpose of integrating XR technologies into the manufacturing process is to provide users with a means of interacting with the robot, facilitating and expediting human-machine communication. Additionally, it aims to offer users a platform for interacting with the simulated robot to enhance learning while mitigating potential risks. In this context, the XR headset (Meta Quest Pro) is utilized to allow users to choose a perimeter on a surface where a collaborative robot will perform a polishing operation. The first phase of the operation involves identifying the surface suitable for operation and presenting it to the user. Subsequently, the user has the ability to pinpoint the exact area for polishing by outlining the contour of the area on the surface. Using extended reality, the user outlines the area on the surface by bringing their index and thumb together, mimicking a pinching gesture. The linedrawing process begins as soon as both fingers touch, creating a virtual pencil that appears between the fingers, simulating the act of drawing on the surface. This operation is intuitively reinforced, as the virtual pencil gives the user a sense of physically painting on the surface. The process concludes when the fingers are separated, causing the virtual pencil to disappear. If the operator repeats the operation, the previous contour line disappears, allowing for the creation of a new one. This flexibility provides the opportunity to redraw the line as many times as needed (Figure 5). Software Architecture and Human-Centric Design Methodology 201 Figure 5: Drawing the line with hand gestures in XR. Training and Upskilling in Simulation In CONVERGING, training and upskilling is provided using the simulation developed in VC4.0 Premium. VC 4.0 Premium provides simulation that can be further developed and used for training and upskilling humans. The workers can be introduced to their working environment early in the design phase of the production cell or manufacturing facility. Workers can immerse themselves inside the environment using VR glasses. Training materials with instructions can be added along with interactive HMI. The final training can be carried out in devices that are based on different operating systems. Visual Components provide free software and mobile application called Visual Components Experience that can be used to view simulation in mobile devices, web and perform VR streaming illustrated in Figure 5. The exploitation of these industry ready digital technologies can be both used for training and upskilling. In the era of labor shortages, upskilling plays a key role, workers from heavy manual work manufacturing sites can be trained for automated factory in digital environments even when the factories of the future are being designed and built. Designers and builders can tap into the tacit knowledge of workers and worker feels more knowhow on the upcoming changes and prepare themselves for new way of working in advance increasing overall productivity. Figure 6: VC4.0 premium simulation export and stream to other platforms using visual components experience. 202 Koukas et al. Contextual Interfaces The Contextual human machine Interfaces are responsible to support the operator and help coordinate the human machine interaction. The interfaces can be personalized in different aspects specifically the contextual interfaces are capable to display different color profiles and also different levels of information based on the user preferences. Figure 6 shows different color palettes of the AR user interfaces, overlayed over the same manufacturing environment. Figure 7: An example on the user interface being overlayed on the manufacturing environment. The interfaces are created as an augmented Reality software suite and aim at supporting the operator’s interaction with flexible mobile robot workers. The developed AR suite, integrated with the shopfloor’s Digital Twin, provides the operators’ a) virtual interfaces to easily program the mobile robots b) information on the process and the production status, c) safety awareness by superimposing the active safety zones around the robots and d) instructions for recovering the system from unexpected events. User Experience and Ergonomy The robot’s pose and motion characteristics affect the operator’s ergonomics in HRC applications; both physical ergonomics and cognitive factors. To enable human-centered robotic motion, we directly integrate models of human ergonomics and human-like motion to the motion planning of the robot. For the ergonomics, we take standard models for ergonomics (RULA, REBA, ISO 11226, DIN EN 1005-4) and write them in an analytical form as a function of the human pose. Then, given a measured, simulated, or potential human pose, we can directly score the upper-torso ergonomics. This enables continuous evaluation of the upper-limb ergonomics, and given a working Software Architecture and Human-Centric Design Methodology 203 pose between human and robot, can enable optimization of robot pose to improve ergonomics. Figure 8: Overview of proposed ergonomics module, with the inputs shown in yellow on the left, and the outputs for the execution and evaluation on the right. An overview of the implementation can be seen in Figure 8, where the human pose, kinematic model, and task parameters are assumed to be available in a ROS network, which are then used to evaluate the ergonomic model and enable optimization of the robot pose. The results are then published for the real-time feedback and the optimized robot pose can be passed to the robot for execution. The visualization can be seen in Figure 9, where the human is in an over-head working position which results in higher ergonomic costs. Figure 9: Visualization of the simulated human upper torso (red) and the resulting ergonomics score as a time plot (bottom).