D1.1 - Real world scenarios and metrics for validation definition

This project has received funding from the European Union’s Horizon Europe - Research and Innovation program under the grant agreement No 101092100. This report reflects only the author’s view and the Commission is not responsible for any use that may be made of the information it contains. D1.1 Real world scenarios and metrics for validation definition Deliverable ID: D1.1 Project Acronym: HARTU Grant: 101092100 Call: HORIZON-CL4-2022-TWIN-TRANSITION-01 Project Coordinator: TEKNIKER Work Package: WP1 Deliverable Type: R-Document, report Responsible Partner: FMI Contributors: ALL Edition date: 30 June 2023 Version: 07 Status: Final Classification: [PU] D1.1 Real world scenarios and metrics for validation definition 2 HARTU Consortium HARTU “Handling with AI-enhanced Robotic Technologies for flexible manufactUring” (Contract No. 101092100) is a collaborative project within the Horizon Europe – Research and Innovation program (HORIZON-CL4-2022-TWIN-TRANSITION-01-04). The consortium members are: 1 FUNDACION TEKNIKER (TEK) 20600 Gipuzkoa | Spain Contact: Iñaki Maurtua [email protected] 2 DEUTSCHES FORSCHUNGSZENTRUM FUER KUENSTLICHE INTELLIGENZ GMBH (DFKI) 67663 Kaiserslautern | Germany Contact: Dennis Mronga [email protected] 3 ASOCIACIÓN DE INVESTIGACIÓN METALÚRGICA DEL NOROESTE (AIMEN) 36418 Pontevedra| Spain Contact: Jawad Masood jawad.maso[email protected] 4 ENGINEERING INGEGNERIA INFORMATICA S.P.A. (ENG) 00144 Rome| Italy Contact: Riccardo Zanetti riccardo.zanet[email protected] 5 TOFAS TURK OTOMOBIL FABRIKASI ANONIM SIRKETI (TOFAS) 34394 Istanbul | Turkey Contact: Nuri Ertekin [email protected] 6 PHILIPS CONSUMER LIFESTYLE BV (PCL) 5656 AG Eindhoven | Netherlands Contact: Erik Koehorst [email protected] 7 ULMA MANUTENCION S. COOP. (ULMA) 20560 Gipuzkoa | Spain Contact: Leire Zubia [email protected] 8 DEEP BLUE Srl (DBL) 00193 ROME | Italy Contact: Erica Vannucci [email protected] 9 FMI HTS DRACHTEN B.V. (FMI) NL-4622 RD Bergen Op Zoom, Netherlands Contact: Floris Goet [email protected] 10 TECNOALIMENTI S.C.p.A (TCA) 20124 Milano | Italy Contact: Marianna Faraldi [email protected] 11 POLITECNICO DI BARI (POLIBA) 70126 Bari | Italy Contact: Giuseppe Carbone giuseppe.carb[email protected]t 12 OMNIGRASP S.r.l. (OMNI) 70124 Bari | Italy Contact: Vito Cacucciolo [email protected] 13 INDUSTRIAL TECHNOLOGY RESEARCH INSTITUTE INCORPORATED (ITRI) 310401 Hsinchu | Taiwan Contact: Curtis Kuan [email protected].tw 14 INFAR INDUSTRIAL Co., Ltd (INFAR) 504 Chang-hua County | Taiwan Contact: Simon Chen [email protected] D1.1 Real world scenarios and metrics for validation definition 3 Document history Date Version Status Author Description 15/05/2023 01 Draft FMI Merged individual use case descriptions 15/06/2023 04 Draft FMI Reviewed all edits. Added company info from proposal to intro section of use cases. 21/06/2023 05 Draft FMI Reviewed structure for requirements and risks. 26/06/2023 06 Draft TEK Reviewed. Final contributions requested 30/06/2023 07 Draft TEK Final submitted D1.1 Real world scenarios and metrics for validation definition 4 Public Executive Summary HARTU sets out to deploy new technologies in different relevant industrial use-cases. This document has been compiled based on input from each of the consortium members and describes the scenarios of 5 industrial partners in more detail: • UC1 – TOFAS – Spare parts delivery preparation • UC2 – TOFAS – Kitting and pre-assembly • UC3 – PCL – Handling for mass customization in the consumer goods sector • UC4 – TCA – Packaging operation in food sector • UC5 – INFAR – Fixtureless assembly in hand tool manufacturing sector • UC6 – ULMA – Pallet to pallet order preparation • UC7 – ULMA – Box to box order preparation Each use case is individually described in detail in chapters 2 to 8. A systems engineering approach was used to determine the requirements of the planned demonstrator for each use case. As such, each chapter is structured in a similar way. • Section 1: Introduction • Section 2: Use case description • Section 3: Demonstrator design • Section 4: Technical risks The common and individual requirements are presented in Section 9. Finally, the result of a very preliminary Risk Assessment in presented Section 10. . D1.1 Real world scenarios and metrics for validation definition 5 Table of contents Table of contents .................................................................................................................................... 5 1 Introduction ............................................................................................................................... 11 1.1 Industrial use-cases ............................................................................................................. 11 1.2 Structure of the document .................................................................................................. 11 1.3 Ethics and human factors .................................................................................................... 12 2 TOFAS-1: Spare parts delivery preparation ............................................................................... 14 2.1 Introduction ......................................................................................................................... 14 2.2 Use case description ............................................................................................................ 14 2.2.1 Initial state ................................................................................................................... 14 2.2.2 Product specifications .................................................................................................. 15 2.2.3 Key figures .................................................................................................................... 18 2.2.4 Target state .................................................................................................................. 18 2.3 Demonstrator design ........................................................................................................... 19 2.3.1 System description ....................................................................................................... 19 2.3.2 Components of the system .......................................................................................... 20 2.3.3 System deployment ..................................................................................................... 21 2.4 Technical risks...................................................................................................................... 22 3 TOFAS-2: Kitting and pre-assembly in the automotive sector ................................................... 24 3.1 Introduction ......................................................................................................................... 24 3.2 Use case description ............................................................................................................ 25 3.2.1 Initial state ................................................................................................................... 25 3.2.2 Product specifications .................................................................................................. 29 3.2.3 Key figures .................................................................................................................... 31 3.2.4 Target state .................................................................................................................. 31 3.3 Demonstrator design ........................................................................................................... 32 3.3.1 System description ....................................................................................................... 32 3.3.2 Components of the system .......................................................................................... 32 3.3.3 System deployment ..................................................................................................... 33 3.4 Technical risks...................................................................................................................... 34 3.4.1 Kitting preparation ....................................................................................................... 34 3.4.2 Assembly operation ..................................................................................................... 35 4 PCL: Handling for mass customization in the consumer goods sector ...................................... 36 D1.1 Real world scenarios and metrics for validation definition 6 4.1 Introduction ......................................................................................................................... 36 4.2 Use case description ............................................................................................................ 36 4.2.1 Initial state ................................................................................................................... 36 4.2.2 Product specifications .................................................................................................. 38 4.2.3 Key figures .................................................................................................................... 39 4.2.4 Target state .................................................................................................................. 40 4.3 Demonstrator design. .......................................................................................................... 40 4.3.1 System description ....................................................................................................... 40 4.3.2 Components of the system .......................................................................................... 43 4.3.3 System deployment ..................................................................................................... 44 4.4 Technical Risks ..................................................................................................................... 44 5 TCA: Sorting operations in agri-food sector............................................................................... 45 5.1 Introduction ......................................................................................................................... 45 5.2 Use case description ............................................................................................................ 45 5.2.1 Initial state ................................................................................................................... 46 5.2.2 Product specifications .................................................................................................. 52 5.2.3 Key figures .................................................................................................................... 56 5.2.4 Target state .................................................................................................................. 57 5.3 Demonstrator design ........................................................................................................... 57 5.3.1 System description ....................................................................................................... 57 5.3.2 Components of the system .......................................................................................... 58 5.3.3 System deployment ..................................................................................................... 61 5.4 Technical risks...................................................................................................................... 61 6 INFAR: Fixtureless assembly in hand tool manufacturing sector .............................................. 62 6.1 Introduction ......................................................................................................................... 62 6.2 Use case description ............................................................................................................ 62 6.2.1 Initial state ................................................................................................................... 62 6.2.2 Product specifications .................................................................................................. 66 6.2.3 Key figures .................................................................................................................... 68 6.2.4 Target state ................................................................................................................. 68 6.3 Demonstrator design ........................................................................................................... 68 6.3.1 System description ....................................................................................................... 68 6.3.2 Components of the system .......................................................................................... 69 D1.1 Real world scenarios and metrics for validation definition 7 6.3.3 System deployment ..................................................................................................... 70 6.4 Technical risks...................................................................................................................... 70 7 ULMA-1: Order preparation: pallet to pallet ............................................................................. 71 7.1 Introduction ......................................................................................................................... 71 7.2 Use case description ............................................................................................................ 71 7.2.1 Initial state ................................................................................................................... 71 7.2.2 Product specifications .................................................................................................. 74 7.2.3 Key figures .................................................................................................................... 74 7.3 Demonstrator design ........................................................................................................... 75 7.3.1 System description ....................................................................................................... 75 7.3.2 Components of the system .......................................................................................... 76 7.3.3 System deployment ..................................................................................................... 77 7.4 Technical risks...................................................................................................................... 77 8 ULMA-2: Order preparation: box to box .................................................................................... 78 8.1 Introduction ......................................................................................................................... 78 8.2 Use case description ............................................................................................................ 78 8.2.1 Initial state ................................................................................................................... 78 8.2.2 Product specifications .................................................................................................. 80 8.2.3 Key figures .................................................................................................................... 80 8.2.4 Target state .................................................................................................................. 80 8.3 Demonstrator design ........................................................................................................... 81 8.3.1 System description ....................................................................................................... 81 8.3.2 Components of the system .......................................................................................... 81 8.3.3 System deployment ..................................................................................................... 83 8.4 Technical risks...................................................................................................................... 83 9 Requirements ............................................................................................................................. 84 9.1 User requirements .............................................................................................................. 84 9.2 Functional requirements ..................................................................................................... 84 9.3 Non functional requirements .............................................................................................. 86 10 Preliminary Risk assessment ...................................................................................................... 88 D1.1 Real world scenarios and metrics for validation definition 8 List of figures Figure 1. Input box preparation in the warehouse (1) and (2) .......................................................... 14 Figure 2. Order preparation at the workshop ................................................................................... 15 Figure 3. TOFAS order preparation overview .................................................................................... 16 Figure 4. Inbound box dimensions (TOFAS) ....................................................................................... 16 Figure 5. Standard outbound box dimensions (TOFAS) ..................................................................... 16 Figure 6. Different outbound boxes, including small cardboard boxes ............................................. 17 Figure 7. Spare parts inside cardboard output boxes ........................................................................ 17 Figure 8. Spare parts inside big, standard output boxes ................................................................... 17 Figure 9. Example of different plastic bags used for packaging ........................................................ 18 Figure 10. Layout of the proposed preparation area (TOFAS) .......................................................... 20 Figure 11. Simulation of TOFAS spare part order preparation scenario .......................................... 20 Figure 12. OnRobot VG vacuum gripper in its 3 possible manual configurations ............................. 21 Figure 13. Order preparation Lab setup (V1) at TEK .......................................................................... 22 Figure 14. Order preparation Lab setup (V2) at TEK .......................................................................... 22 Figure 15. Kit preparation, transport and assembly ......................................................................... 24 Figure 16. Products in special compartments ................................................................................... 24 Figure 17. Products in semi-structured configuration ....................................................................... 24 Figure 18. Products randomly distributed ......................................................................................... 24 Figure 19. The Process Flow of The Rear Wheel Production Line ..................................................... 25 Figure 20. The Layout of kitting and assembly processes ................................................................. 26 Figure 21. Kitting preparation area. ................................................................................................... 26 Figure 22. Destination containers on the conveyor. The blue boxes contain the discs .................... 26 Figure 23. Push buttons to control the conveyor belt that transports the output containers ......... 27 Figure 24. Pick to light device above each input container ............................................................... 27 Figure 25. Operator taking a component from the input container (left side). It will then put on the destination container (right side) ...................................................................................................... 27 Figure 26. Hub .................................................................................................................................... 30 Figure 27. Brake plate ........................................................................................................................ 30 Figure 28. Spindle ............................................................................................................................... 30 Figure 29. Heat cover ......................................................................................................................... 30 Figure 30. Brake disc .......................................................................................................................... 30 Figure 31. Drum ................................................................................................................................. 30 Figure 32. Brake caliper...................................................................................................................... 31 Figure 33. Proposed demonstrator for the kitting operation ............................................................ 33 Figure 34. Spindle containers: one on top of each other .................................................................. 34 Figure 35. Current design of the output container ............................................................................ 34 Figure 36. Step by step breakdown of the PCL lacquering process ................................................... 36 Figure 37. Lacquering process overview ............................................................................................ 37 Figure 38. Loading and unloading area .............................................................................................. 37 Figure 39. Chest panel ........................................................................................................................ 38 D1.1 Real world scenarios and metrics for validation definition 9 Figure 40. Front panel ........................................................................................................................ 38 Figure 41. Chest jig ............................................................................................................................. 39 Figure 42. Front panel jig ................................................................................................................... 39 Figure 43. Trays dimensions ............................................................................................................... 40 Figure 44. Different gripping alternatives .......................................................................................... 41 Figure 45. Example of a tray that does not allow gripping from the inside of the workpiece (left) . 41 Figure 46. Proposed layout for PCL use case ..................................................................................... 42 Figure 47. Component in the proposed system for PCL use case ..................................................... 43 Figure 48. Agri-food sector: scheme of activities .............................................................................. 45 Figure 49. Location and production area of Centrolazio: (left) Map of Central Italy, (right) in light blue the greenhouses, in green the processing area. ........................................................................ 46 Figure 50. Seasonality of zucchini, eggplant and tomato .................................................................. 46 Figure 51. Line 1 in Centro Lazio for zucchinis and some type of tomatoes. .................................... 47 Figure 52. Setup of Line 1 of Centro Lazio ......................................................................................... 48 Figure 53. Line 1 workbench. On the left the blue box with unsorted products. The black boxes are for the sorted product ....................................................................................................................... 48 Figure 54. Setup of Line 2 of Centro Lazio ......................................................................................... 49 Figure 55. Conveyor belt and workbench of Line 2 in Centro Lazio .................................................. 49 Figure 56. Overview of the processing area ...................................................................................... 51 Figure 57.Output of processing lines: boxes with products of similar quality/size ........................... 52 Figure 58. Zucchinis ............................................................................................................................ 52 Figure 59. Eggplant ............................................................................................................................. 54 Figure 60. Tomatoes .......................................................................................................................... 55 Figure 61. TCA demonstrator overview ............................................................................................. 58 Figure 62. Initial design of the TCA demonstrator ............................................................................. 59 Figure 63. Footprint of the TCA demonstrator .................................................................................. 59 Figure 64. Safety monitoring areas covered by the two lasers (left) or four radars (right) .............. 60 Figure 65. Main components of the wrench ..................................................................................... 62 Figure 66. Assembly workshoop overview ........................................................................................ 63 Figure 67. Operators at the assembly tables ..................................................................................... 63 Figure 68. INFAR: Selected operations .............................................................................................. 67 Figure 69. Parts involved in the INFAR use case ................................................................................ 67 Figure 70. Hand tools market............................................................................................................. 68 Figure 71. Proposed The Robot Cell Controller for multi-robotic assembly ..................................... 69 Figure 72. From the exploded-view drawing to assembly completeness via robotic assembly process ............................................................................................................................................... 69 Figure 73. Real example of output pallet at BESA ............................................................................. 72 Figure 74. Operator manually placing a box on the pallet ................................................................ 72 Figure 75.Operator placing a can on the pallet with the help of a material handling crane ............ 72 Figure 76. Pallet to pallet order preparation workflow ..................................................................... 73 Figure 77. Examples of products manipulated in a pallet to pallet scenario .................................... 74 Figure 78. ULMA palletizing use case ................................................................................................. 75 Figure 79. Product types in the ULMA palletizing scenario ............................................................... 75 Figure 80. ULMA Palletizing station design ........................................................................................ 76 D1.1 Real world scenarios and metrics for validation definition 16 Figure 3. TOFAS order preparation overview The two types of boxes (input and output) have the following dimensions: Figure 4. Inbound box dimensions (TOFAS) Figure 5. Standard outbound box dimensions (TOFAS) D1.1 Real world scenarios and metrics for validation definition 17 Figure 6. Different outbound boxes, including small cardboard boxes Figure 7. Spare parts inside cardboard output boxes Figure 8. Spare parts inside big, standard output boxes D1.1 Real world scenarios and metrics for validation definition 18 2.2.3 Key figures Key figures: ▪ 63.000 spare parts. ▪ Average cycle time: 45 seconds/part. ▪ The error rate is 0,03 % of the total (loading of an incorrect part into the output boxes). 2.2.4 Target state Based on the experience gained in the Horizon 2020 PICKPLACE project these constraints are introduced: 1. Inbound boxes shall only contain products that come in cardboard boxes. The current procedure includes the mixing of any kind of products in the same inbound box. This means that products in cardboard boxes, products in plastics bags and unpacked products are included in the same box. The presence of plastic bags represents a serious difficulty for grasping, either by vacuum or with 2-3 fingers, as there is no way of knowing the shape of the product inside and, depending on the plastic used, the vacuum doesn’t work. However, 60% of products come in carboard boxes. The conclusion in PICKPLACE was that by adapting the preparation procedure in the warehouse (carboard boxes in one inbound box and those coming in plastics bags and those unpacked in another), and creating a collaborative application on the preparation shopfloor, it was possible to achieve an efficient system. 2. To use a mobile manipulator to handle the cardboard boxes. In PICKPLACE, a robotic arm mounted on a linear axis was used to deliver the products to the different outbound boxes. This was sufficient to validate the results on a lab scale, but is not suitable for a real industrial setting, due to (1) the lack of flexibility to manage many different configurations of input/outbound boxes, (2) creates physical barriers that hinder the movements of human workers, and (3) is a costly solution. Instead, in HARTU, it is proposed to use a mobile platform with a robotic arm mounted on it Figure 9. Example of different plastic bags used for packaging D1.1 Real world scenarios and metrics for validation definition 19 The proposed approach will improve the working conditions for human operators, reduce the number of errors and increase the efficiency of the system. 2.3 Demonstrator design 2.3.1 System description As explained in previous section, in HARTU, it is proposed to use a mobile platform with a robotic arm mounted on it, which will be in charge of handling spare parts packaged in carboard boxes. The proposed procedure is as follows: 1. The platform approaches the inbound box area. 2. It takes an image of the inside of the box and identifies the most suitable product to be picked up. o If needed, the platform will navigate to a different position to reach the product. 3. The arm movement is planned and executed to pick the target product. 4. The robot identifies the reference of the picked product by means of the bar code labelled on the carboard and sends the information to the WMS. o It may involve the robot presenting the different sides of the product to the barcode reader device (depending on the position of the bar code label on the box) o The design of a workstation in charge of this identification will be considered. 5. The robot navigates to the position of the destination output box (this information is provided by the warehouse management system) 6. The robot calculates the trajectory to place the product in the right position (creating a mosaic to optimize occupancy) 7. Once the part has been placed, the robot takes an image of the inside of the box to monitor the status of the products inside. This information will be used for the placement of the next product (step 6). The following pictures present the new concept. D1.1 Real world scenarios and metrics for validation definition 20 Figure 10. Layout of the proposed preparation area (TOFAS) Figure 11. Simulation of TOFAS spare part order preparation scenario 2.3.2 Components of the system • Mobile manipulator, consisting of: o Segway RMP omnidirectional mobile robot o 7-axis KUKA iiwa (14 kg payload) The dimensions of the base are: W = 788 mm, L =1350 mm, H = 897 mm. • RGB-D or Photoneo Camera mounted on an eye-in-hand configuration on the robot. • Gripper D1.1 Real world scenarios and metrics for validation definition 21 o Based on suction principle. Due to the mobile platform we plan to use, the best option would be to use a gripper with an electrically powered gripper, so no external air supply is needed. A possible one is the VG10 from OnRobot, that has these dimensions: Folded: 146 x 146 x 102 mm. Unfolded: 390 x 390 x 102 mm Figure 12. OnRobot VG vacuum gripper in its 3 possible manual configurations Other alternatives will be analysed, including some small form factor air supply tools, such as PIAB Kenos KCS, Coval CVGC150X150, or FORMHAND FH-R80. • Dual gripper. Due to the different sizes of the boxes, it might be needed a tool exchange unit, or the use of a dual gripper (or both). The change of tool can be done in the way back from the release operation to the new picking (as far as the image of the input box is taken after each picking). An alternative can be to adapt the warehouse management, so the input boxes include references of similar sizes. In this case the operator in the preparation area will verify the appropriate gripper is on the robot each time a new input box arrives. • Part identification (barcode). Two alternatives: o A bar code reader is placed on the mobile platform. As the label can be placed on any of the 6 sides of the box, the robot will move the box in front of it (to place the label in the reader’s field of view. o To design a device that automatically identifies the box and includes a camera to identify the position of the part before regrasping it. This device can be used in ULMA box-to-box use case as well. 2.3.3 System deployment The platform is provided by TEKNIKER, which will be in charge of system integration. The system will be tested in TEKNIKER’s facilities and finally transported to Bursa to validate in the real TOFAS facilities. The lab setup will include one inbound box and 1 to 3 outbound boxes, due to space limitations at TEK shopfloor. D1.1 Real world scenarios and metrics for validation definition 22 Figure 13. Order preparation Lab setup (V1) at TEK Figure 14. Order preparation Lab setup (V2) at TEK The green circles in the figures above represent the possible barcode reading, pose estimation and re-grasping device to be designed. The two placement alternatives are presented: mounted on the robot or mounted on the workshop floor. The first option is the preferred, but space constraints need to be analysed. 2.4 Technical risks . D1.1 Real world scenarios and metrics for validation definition 23 CODE Description Probability Impact R.1 Carboard boxes flaps are not properly positioned. Mitigation measure: If the robot detects this situation, the operator will be informed, and the robot will stop the process until the problem is solved Low (2) Medium (3) Very Low=1; Low =2; Medium = 3; High=4; Very High=5 D1.1 Real world scenarios and metrics for validation definition 24 3 TOFAS-2: Kitting and pre-assembly in the automotive sector 3.1 Introduction This use case corresponds to the preparation of kits of components (operation known as ‘kitting’ in the automotive sector) ant their pre-assembly at the corresponding assembly workstation. At TOFAS there are 120 kitting areas where human workers prepare kits of products that are then delivered to other areas of the factory to be assembled into sub-assemblies or into the final car. In the kitting area, products are taken from containers/boxes in which they can be arranged in three main configurations: (1) Product-specific individual compartments (Figure 16); (2) Semi-structured configuration (Figure 17), forming layers that are separated by means of separators (cardboard or plastic); (3) randomly distributed (Figure 18). The kits, composed of products with different geometries, materials and dimensions, are placed on carriers that are towed by conveyor belts and/or AGVs through the factory to the destination. The kits, once at the assembly workstation, are downloaded and the parts assembled manually, mainly using insertion and screwing techniques with the support of hand tools and devices. Figure 16. Products in special compartments Figure 17. Products in semi-structured configuration Figure 18. Products randomly distributed Figure 15. Kit preparation, transport and assembly D1.1 Real world scenarios and metrics for validation definition 25 3.2 Use case description The use case selected in HARTU is the Real Wheel 356 kitting and assembly, internally identified at TOFAS as "Rear Wheel 356 - Drum - OP30" and "Rear Wheel 356 - Disc - OP30”. 3.2.1 Initial state 3.2.1.1 Process The production line consists of three workstations, where two operators assemble the rear wheels using mainly insertion and screwing techniques with the support of hand tools and devices. These workstations are identified as: • Op 10: Rear wheel 356-Drum and Rear wheel 356-Disc • Op 20: Rear wheel 356-Drum • Op 30: Rear wheel 356-Drum and Rear wheel 356-Disc The process workflow of the rear wheel production line and the schematic layout of the area can be seen in Figure 19. The current setup consists of 2 main areas: one for kitting (it is done in two sub-areas, one for OP10 components and the other one for OP20 and OP30), and one assembly area with three workstations (OP10, OP20 and OP30). Figure 19. The Process Flow of The Rear Wheel Production Line D1.1 Real world scenarios and metrics for validation definition 32 Given the diversity of assembly tasks, the operations are to be performed either by a human or by a robot. The concrete division of tasks between human and robot is still to be defined. It depends on the maximum robot speed, the workspace layout, and the complexity/ergonomics of the tasks. Ideally the robot takes over heavy and tedious tasks without disturbing the human or slowing down the overall workflow. Collaborative assembly is an option for heavy objects, e.g., metallic cylinder (drum). The robot is either mounted in a fixed position next to the assembly line or on a mobile base. The latter is preferred as it allows the robot to navigate between the assembly stations, increase its flexibility, and enlarges the reachable workspace. 3.3 Demonstrator design 3.3.1 System description As mentioned above, the use case corresponds to a common activity in car manufacturing plants: the preparation of component kits of that need to be assembled prior to be fitted to the car, or that are assembled directly to the car. In HARTU, the rear wheel assembly kit preparation and the rear wheel drum assembly (Rear Wheel 356 - Drum - OP30) have been selected. 3.3.2 Components of the system 3.3.2.1 Kitting preparation The proposed system will use the same robotic platform as in the order preparation use case. It consists of: • Mobile platform, consisting of: o Segway RMP omnidirectional mobile robot o 7-axis KUKA iiwa (14 kg payload) The dimensions of the base are: W = 788 mm, L =1350 mm, H = 897 mm. In case of need, we can adopt the MIR200+UR10 configuration. • RGB-D or Photoneo Camera mounted on an eye-in-hand configuration on the robot. • Gripper: Based on magnetic principle or fingers. In both cases, electrically actuated The procedure to be followed is as follows: • The operator press the conveyor belt control button to request the arrival of the 10 empty output containers. • The robot receives the list of products and the order in which they have to be placed in the output containers from the WMS. • The robot takes a picture of each box and starts feeding the part: • The platform approaches to the corresponding inbound box. • It takes an image of the interior and identifies the most suitable product to be picked. • It plans and executes the arm movement to pick up the target product. D1.1 Real world scenarios and metrics for validation definition 33 • The robot navigates to the position of the destination container box on the. conveyor belt • The robot calculates and executes the trajectory to place the product in the right position. • In case of having to remove a separator between layers, it requests the help of a human operator from the assembly area. • These steps are repeated until all items in the list have been placed in the output containers. • Once all parts are handled by the robot, it requests the presence of the human operator who introduces the discs and the 10 paper forms. • Finally, the operator press the conveyor button to transport the filled boxes to the assembly area. 3.3.2.2 Assembly operation The proposed demonstrator uses the same robotic platform as the kitting preparation. The platform is placed next to the assembly line (opposite to the human operator). Together with the human worker, the robot performs the assembly of the rear wheel (Rear Wheel 356 - Drum - OP30). Thereby, human and robot work in a shared workspace. Ideally the workflow is smooth and efficient, i.e., the robot should not disturb the human worked or slow down the assembly process. 3.3.3 System deployment 3.3.3.1 Kitting preparation The platform is provided by TEKNIKER, which will be in charge of the integration. The platform will be tested in TEKNIKER’s facilities and finally transported to Bursa to validate in the real TOFAS facilities. The lab setup will include inbound boxes and outbound boxes on a table (instead of the conveyor). Figure 33. Proposed demonstrator for the kitting operation D1.1 Real world scenarios and metrics for validation definition 34 3.3.3.2 Assembly operation An initial laboratory demonstrator will be prepared at DFKI Robotics Innovation Center in Bremen, Germany. The system will comprise an industrial robotic manipulator (Kuka iiwa) with 3-finger robotic gripper, as well as a mock-up of the assembly line. The final setup will comprise a mobile platform with Kuka iiwa robot provided by TEK. It will be evaluated at the TOFAS assembly line, without interfering with actual production. 3.4 Technical risks 3.4.1 Kitting preparation CODE Description Probability Impact R.1 Currently two containers are placed on top of each other (see Figure 34). the robot will not be able to reach the lower container. Mitigation measure: layout with containers in parallel. High (4) High (4) R.2 The current output container is designed for manual placement of the objects (see Figure 35), but it will be too complex for robotic operation. The mitigation measure will be the redesign of the container (no problem for the rest of the line and the workstations) Medium (3) High (4) Very Low=1; Low =2; Medium = 3; High=4; Very High=5 Figure 34. Spindle containers: one on top of each other Figure 35. Current design of the output container D1.1 Real world scenarios and metrics for validation definition 35 3.4.2 Assembly operation CODE Description Probability Impact R.1 The Parts are too diverse to be handled by a single robotic gripper. Mitigation measure: Introduce a tool changer or leave the parts that cannot be handled by the robot to the human worker. High (4) Low (2) R.2 Precision (especially the positioning of the part inside the robot gripper) is too low for robotic assembly (e.g., pre-screwing). Mitigation measures: Use robot compliance to account for small inaccuracies, use sensor to estimate the position of the part inside the gripper, use object pose estimation to allow precise and repeatable grasping. High (4) Medium (3) R.3 Parts are too heavy to handle with a single robot gripper. Mitigation measure: Handle large parts in collaboration with the human. Low (2) Low (2) Very Low=1; Low =2; Medium = 3; High=4; Very High=5 D1.1 Real world scenarios and metrics for validation definition 36 4 PCL: Handling for mass customization in the consumer goods sector 4.1 Introduction PHILIPS is world leader in mass production of consumer goods. In the business of consumer goods there is more and more demand for personalized products. Traditionally, consumer products are mass-produced. Products which are mass-produced are optimized for efficient production, meaning limited product variance and high production volumes to keep the factory cost price low and the product available for everyone. Personalization means an extra effort needed to personalize the product. Today, that would mean more manual labor in a lot of cases. This conflicts with the aim to keep products within reach of everyone. 4.2 Use case description One area where diversity is added is the workpiece lacquering line, where parts are coated with a layer of lacquer to match the product design to the consumer’s need. Up to now, this has been done by manual labor. There is a strong wish to automate this process and this production step will be the subject of this use-case. 4.2.1 Initial state Typical production line components which rely heavily on standardization are robotic arms and their grippers. Currently, robots need a highly structured workplace to be able to do their work. In a lot of situations, it is possible to comply with this need. In the lacquering line however, this is not possible due to the flexibility required (many different products). Motions are now designed for the human hand and with current robot technology it will be very difficult to automate this process. The aim of the HARTU project is therefore to investigate new technologies to make the next step in robotization. 4.2.1.1 Process A step-by-step description of the lacquering process is shown in figure: Although the complete process of applying lacquer to shaver parts includes many steps, in HARTU we focus on the physical aspects of placing the parts on the jig and taking them off again after the lacquer is applied. Preparing input parts Transfer parts from tray to jig Apply lacquer to parts Transfer parts from jig to tray Figure 36. Step by step breakdown of the PCL lacquering process D1.1 Real world scenarios and metrics for validation definition 37 An overview of the lacquering line is given in next figures. Figure 37. Lacquering process overview Figure 38. Loading and unloading area In this links: HARTU-PCL-Chest-1.MP4, HARTU-PCL-Chest-2.MP4, HARTU-PCL-FrontPanel-1.MP4 and HARTU-PCL-FrontPanel-2.MP4. D1.1 Real world scenarios and metrics for validation definition 38 4.2.1.2 Inputs Process input definitions Environmental conditions: ▪ No specific demands for temperature, humidity, etc. ▪ Any unexpected mechanical situation is handled by human operators, so worst case mechanical conditions are not known. Personnel: ▪ The lacquering line is operated by a team leader and 5 operators. ▪ The skill level of operators is secondary school. Materials in: ▪ Different geometries of molded plastic parts ▪ Lacquering jigs Supplies in: ▪ Air for pneumatics ▪ Electrical (400 V) 4.2.1.3 Outputs Process output definitions Materials out: ▪ Lacquered and quality inspected parts ▪ Scrap parts ▪ Reusable jigs Information out: ▪ Yield and scrap figure ▪ Batch info 4.2.2 Product specifications Two plastic outer shell parts are identified for this use case: • Chest • Front panel A detailed overview of the part properties is given in HARTU-PartsCharacterization.xlsx. The plain plastic input parts and the jigs used for fixturing the parts are shown in next figures. Figure 39. Chest panel Figure 40. Front panel D1.1 Real world scenarios and metrics for validation definition 39 Figure 41. Chest jig Figure 42. Front panel jig The pictures show an intermediate stage of the process when the parts are placed on the jigs. After the entire process is completed, the parts are again in the trays, as shown in next picture. 4.2.3 Key figures • 6 parts per spindle • 500 spindles on the chain • 8 seconds cycle time for each advancement of the chain • 3 loading positions • 2 unloading positions • 2 products are (un)loaded to or from tray in one cycle D1.1 Real world scenarios and metrics for validation definition 40 4.2.4 Target state As already mentioned, the use of manual labor to achieve the required degree of production flexibility is relatively expensive, and it is an opportunity to have robots to do this work. The ideal outcome would be that a robot with a universal gripper is able to take the parts from the tray and put them on the jig and vice versa. This will result in a reduction of the human labor force engaged in repetitive and low value-added tasks, as seen in the videos cited at the end of section 4.2.1.1, while allowing for flexible production systems. This is particularly important in countries suffering from labor shortages. 4.3 Demonstrator design. 4.3.1 System description The scope of the intended system is to demonstrate the placement and removal of products on the lacquering jig. The system shall be designed and validated for the two described product ranges, but not be limited to it. Furthermore, the perception system used for product pose estimation shall cope with color variations. The system will be commissioned in a relevant environment, but not on the actual lacquering line. Therefore, a continuously looping demonstrator can be achieved by using a single tray from which products are picked and to which products are placed back. The supplied tray contains a single type of outer-shell part in a structured way. The parts lay loose within the tray resulting in a positional uncertainty that must be resolved before fixturing. The 3D models of all parts are known and can be used by the grasping point estimator and the vision algorithm for pose estimation. Figure 43. Trays dimensions The parts can be gripped using a regular two-finger gripper, either from the inside or from the outside, depending on the part type and tray type. D1.1 Real world scenarios and metrics for validation definition 41 Figure 44. Different gripping alternatives However, the smaller tray used for chest parts has two variants: one with a support at the center and one without (see Figure 45). Therefore, some chest pieces cannot be gripped from the inside. Figure 45. Example of a tray that does not allow gripping from the inside of the workpiece (left) The expected process flow will be: 1. Set-up 1. Operator supplies filled tray 2. Operator places jig for selected product type on spindle. 3. Operator presses start button on HMI 2. Fixturing D1.1 Real world scenarios and metrics for validation definition 48 Figure 52. Setup of Line 1 of Centro Lazio The workflow is as follows: 1. the operator loads 3 or 4 empty boxes on the workbench (black boxes (1) in Figure 52) to be filled with selected products; 2. takes a coloured box of unsorted zucchinis from the rotating loading belt and places it next to the workbench (coloured box (2) in Figure 52); 3. picks a zucchini from the input box; 4. checks that the zucchini has the right size and shape; 5. checks for defects on the zucchini; 6. places the zucchini in the corresponding black box according to quality and size (1° class, 2° class, 3 class, waste); 7. in case of serious defects, the operator places the zucchini in the waste bin on the left; 8. puts the full black box on the upper conveyor belt for unloading. Figure 53. Line 1 workbench. On the left the blue box with unsorted products. The black boxes are for the sorted product (1) (2) D1.1 Real world scenarios and metrics for validation definition 49 Line 2 (Eggplant) Line 2 is used for eggplant and is characterized by a different working method, including a washing phase at the beginning of the line. The main difference is that the product runs on the belt in bulk, rather than in boxes. The operator at the workstation picks the products from the belt and places them in the box according to their quality and size. Figure 54. Setup of Line 2 of Centro Lazio The workflow is as follows: 1. The operator places the empty boxes on the workbench; 2. takes the item from the conveyor belt. (blue belt in Figure 55); 3. checks the product for defects; 4. places the product without defects in the box; 5. if the product has defects, the operator places it in the waste bin; 6. when the box is full, the operator places it on the output line (upper grey belt in Figure 55) Figure 55. Conveyor belt and workbench of Line 2 in Centro Lazio Output line for full boxes Conveyor belt for unsorted products D1.1 Real world scenarios and metrics for validation definition 50 Categorization of boxes used in the use case Type Material / Dimensions (W x L x H) cm Used for Plastic box: 30 x 50 x 27 Input box from the field used to load the production line. Used for tomatoes and zucchinis Plastic bin: 34.5 x 52 x 30 Input box from the field used to load the production line. Used for eggplants. They will not be considered in the use case Plastic box: 30 x 40 x 15 Output box for sorted products. Usually, black colored Used for zucchinis and eggplants Plastic CPR box: 40 x 60 x 10 Output box for sorted products with collapsible sides. Used for zucchinis and eggplants Cardboard box: different sizes 40 x 60 x 12-14 30 x 40 x 12-14 30 x 50x 12-14 Output box for sorted products. Used for tomatoes Wooden box: different sizes 40 x 60 x 12-14 30 x 40 x 12-14 30 x 50 x 12-14 Output box for sorted products. Used for tomatoes D1.1 Real world scenarios and metrics for validation definition 51 5.2.1.2 Inputs The working environment is an agricultural shed. The shed is an open space, with no physical separation between areas. There are several lines for different food products. There is not any air conditioning control system, therefore, the temperature and humidity of the working environment depends on the outside temperature and humidity in a range that usually varies between 6°C and 30°C, and up to 90% humidity. Processing does not require a clean room. Figure 56. Overview of the processing area The activity of selecting and boxing the product is performed by one person per working station, but a line can have several people working at the same time on different stations (see Figure 51). The operator works close to the line where the raw product is distributed (in coloured boxes or in bulk). The operator picks up a box from the line and places it in a buffer area (the conveyor belt brings the boxes in a circular ring) or picks up the bulk product. The operator must be able to handle the product carefully and quickly with both hands, visually detect any defects on the product, sort and place the product in the correct box in an oriented way until this output box is completely full. Another operator collects the filled boxes and builds the pallet, that is finally weighed and labelled. No specific skills are required, other than the ability to sort the product, for which they rely on training and experience. 5.2.1.3 Outputs The output of the process is a box filled with products. The products are homogeneous and meet the selection criteria. In some cases (For example zucchinis and eggplants), the products are arranged in the box in an oriented way (Figure 57). Raw material boxbox Logistic GATE Line 1 Line 2 Workstation Raw Material (RM) boxes Load of RM boxes Logistic area Selected product box D1.1 Real world scenarios and metrics for validation definition 52 Figure 57.Output of processing lines: boxes with products of similar quality/size Products must not overflow from the box. 5.2.2 Product specifications Regarding the food user case, 3 relevant products have been identified: • Zucchinis; • Eggplants; • Tomatoes. 5.2.2.1 Zucchini The zucchini is a long, cylindrical vegetable, slightly smaller at the stem end, usually dark green in colour. The flesh is a pale greenish-white and has a delicate, almost sweet flavour. The zucchinis fruit grows quickly and is harvested within 2 to 7 days of flowering. Zucchinis must be sorted by category and size according to COMMISSION REGULATION (EC) No 1757/2003 of 3 October 2003: Class Extra - the zucchinis in this category must be of superior quality. They must present the characteristics of the variety and/or commercial type. They must be: ▪ well developed, ▪ well formed, ▪ provided with a peduncle, cut cleanly and not more than 3 cm in length. They must be defect-free. Only very slight superficial defects provided these do not affect the general appearance of the product and its essential characteristics are admissible. Class I - the zucchinis in this class must be of good quality. They must be characteristic of the variety and/or commercial type. The following slight defects, however, may be allowed provided these do not affect the general appearance and quality of the product: Figure 58. Zucchinis D1.1 Real world scenarios and metrics for validation definition 53 • slight shape defects. Zucchinis may be curved, with a curvature of no more than 10° from vertical. Spearhead shape is not allowed. • slight colour defects: small areas with a lighter or darker colour than the rest of the product. • very slight skin defects: small, barely visible scratches. • very slight defects due to diseases provided they are not progressive and do not affect the flesh (presence of small black dots or areas where the has small white spots). The zucchinis must have a stalk not exceeding 3 cm in length. Class II - includes zucchinis which do not qualify for inclusion in the higher classes. They may show the following defects, provided these do not affect the essential characteristics as regards the quality, keeping quality and presentation: • slight shape defects The zucchini may be curved, with a curvature of no more than 20° from vertical. spearhead shape is not allowed) • slight colour defects, (Small areas with a lighter or darker colour than the rest of the product) • very slight skin defects, (small visible scratches) • slight sunburn (small with or yellow stain less than 1 cm in diameter), • very slight defects due to diseases provided they are not progressive and do not affect the flesh. (presence of small black dots or areas where the has small white spots) Size category The size of the zucchinis is determined by both length and weight. Length in cm: between the junction with the stalk and the tip: Categories for the demonstrator: • Cat.1: 7 cm ≤ d ≤ 14 cm • Cat.2: 14 cm < d ≤ 21 cm • Cat.3: 21 cm < d We can assume a measuring tolerance of ±1 cm. Weight in g.: the minimum weight is 50 g. and the maximum weight is 450 g. • from 50 g. to 100 g., • 100 g. to 225 g, • 225 g. to 450 g. 5.2.2.2 Eggplant Fruits are round or elongated in shape, very smooth, with shiny black or very dark purple skin and bright green petiole. Fruit texture and hardness are very high. D1.1 Real world scenarios and metrics for validation definition 54 Eggplants must be: • whole; fresh-looking; firm; clean (free of visible foreign matter); with the calyx and stalk attached; • healthy (products affected by rot or showing changes that make them unsuitable for consumption are excluded); • with bright black epicarp, without reddish tinge, abrasions or other alterations; firm fruit • with non-fibrous or woody flesh and underdeveloped seeds; free of abnormal external moisture; free of strange smell and/or taste. Eggplant Quality Category Class I (good quality) with typical characteristics of the variety, free from sunburn. Slight defect in shape, slight discoloration of the base, slight damages not exceeding 3 cm² of the surface. Class II (with minimum quality characteristics) must exhibit the essential characteristics of quality and presentation. There may be present defects in shape, discoloration, slight sunburn and healed skin defects not exceeding 4 cm² of the surface area. Eggplant Size Category Calibration is determined by diameter (in mm), or weight (in g), and is mandatory for Class I. Classification by diameter • minimum allowed 70 mm. • the difference, in the same box, between the smallest and the largest must not exceed 25 mm. In addition, for long ones, the minimum length of the stalk is 80 mm. For the eggplants used in the demonstrator we will consider these categories: • Cat.1: 70 mm ≤ d < 95 mm • Cat.2: 95 mm ≤ d < 120 mm • Cat.3: 120 mm ≤ d < 145 mm • Cat.4: 145 mm ≤ d Classification by weight • minimum weight 100 gr. • 100 to 300 g with a maximum difference of 75 g between the smallest and largest eggplant within the same box; Figure 59. Eggplant D1.1 Real world scenarios and metrics for validation definition 55 • 300 to 500 g with a maximum difference of 100 g between the smallest and the largest eggplant within the same box; • greater than 500 g with a maximum difference of 250 g between the smallest eggplant and the largest eggplant within the same box. 5.2.2.3 Tomato Characterized by a large fruit, the tomato is consumed when it turns from a deep green to a red coloration. Tomatoes Quality Category • Extra (higher quality): very slight surface defects of the epidermis are allowed. Greenback is not allowed. • Class I (good quality): they must not show cracks or green back. Slight defects in shape, development and coloration, epidermis defects and bruising allowed. • Class II (with minimum quality characteristics): tomatoes must have no unhealed cracks. Allowed slight defects related to shape, development and coloration, epidermis defects or bruises, provided they do not seriously damage the fruit, healed cracks up to 3 cm in length. Tomatoes Size Category The size is determined by the maximum diameter of the section normal to the axis of the fruit. • 30 mm ≤ d < 35 mm • 35 mm ≤ d < 40 mm • 40 mm ≤ d <47 mm • 47 mm ≤ d < 57 mm • 57 mm ≤ d < 67 mm • 67 mm ≤ d < 82 mm • 82 mm ≤ d < 102 mm • 102 mm ≤ d The difference in diameter between tomatoes in the same box is limited to: • 10 mm, if the diameter of the smallest fruit is < 50 mm; • 15 mm, if the diameter of the smallest fruit is ≥ 50 mm < 70 mm; • 20 mm, if the diameter of the smallest fruit is ≥ 70 mm < 100 mm; • no threshold difference is set for tomatoes with a diameter of 100 mm or more. Figure 60. Tomatoes D1.1 Real world scenarios and metrics for validation definition 56 For the tomatoes used in the demonstrator we will consider three different categories, such us: • Cat.1: 30 mm ≤ d < 35 mm • Cat.2: 35 mm ≤ d < 40 mm • Cat.3: 40 mm ≤ d < 47 mm It is accepted that 10% of the products are outside the category dimension. 5.2.3 Key figures Product Cycle time Tolerances Zucchini 2-3 minutes to complete a box of zucchinis (approximately 6-7 kg) Zucchini Size tolerance • The difference in size between the fruits in the same box for all categories is 10% in +/- in number or weight. Zucchini quality tolerance: • Class Extra: up to a maximum 5% by number or weight of products not conforming to the characteristics of the class. • Class I: up to a maximum of 10% by number or weight of product not conforming to the characteristics of the class. • Class II: up to a maximum of 10% in number or weight of product not meeting the characteristics of the class, However, non-standard fruits must be suitable for consumption. Eggplants 1 box of finished product of eggplant processing time 3 minutes. Box weights 10 kg Eggplant size tolerance • Class I, +/- 10% by number or weight of the indicated size. • Class II, +/- 10% by number or weight of the minimum size. • Tolerated product below the minimums: 5 mm and 90 g. Eggplant quality tolerance • Class I: up to a maximum of 10% by number or weight of product not meeting the characteristics of the category. • Class II: up to a maximum of 10% by number or weight of product not meeting the characteristics of the category. Tomatoes 1 box of finished product of tomato: processing time 5 minutes. Box weights 12.5 kg Tomatoes size tolerance The difference in diameter between the fruits in the same box for all categories is 10 % in +/- in number or weight. Tomato quality tolerance • Class Extra: up to a maximum of 5% by number or weight of product not meeting the characteristics of the category. • Class I: up to a maximum of 10% by number or weight of product not meeting the characteristics of the category. • Class II: up to a maximum of 10% by number or weight of product not meeting the characteristics of the category. It is noteworthy the trend to reduce the defects rate to below 1%. D1.1 Real world scenarios and metrics for validation definition 57 5.2.4 Target state Nowadays, picking, sorting and placing activities are carried out by hand by the workers. This is a repetitive work where the products are handled and transferred piece by piece along the line or into the boxes for delivery. This is costly and can lead to unhealthy working conditions and, above all, possible hygienic problems. Also, some products require major attention for not damaging them during this phase, compromising its quality and shelf life. A robot (a cobot or an industrial robot with the corresponding safety measures) that can handle vegetables of different sizes, shapes, weights and softness, even when they are randomly distributed is a wish. The system must be fast and safe for the products and working alongside humans. Because of line process (boxes running on conveyor belt), a human handling the boxes is required for moving unsorted box out of the line (close to the robot) and manage full boxes. Using the HARTU solution a few number of persons will be necessary for coordinating and synchronizing the activity of the system. In line with the objectives of the HARTU project, a simplified user case will be developed to demonstrate the effectiveness of the grasping technology. This use case will use the shape and dimension classification criteria (no defect inspection or weight). 5.3 Demonstrator design 5.3.1 System description The demonstrator will be based on the packaging of zucchinis, eggplants, and tomatoes. It basically consists of a robot picking products from the input box, classifying them according to size and shape and placing them in the corresponding output box. The procedure to be followed is as follows: • The line operator places the input and output boxes in the corresponding station of the demonstrator. • An image is taken from the top of the inbox. The most suitable product to be picked is identified using the grasping point identification component (TEK) • The robot generates the trajectory to pick the selected product and, once picked, moves it to the size/shape inspection area, where a new image is acquired. • The system classifies the product according to size and shape and asks the robot to place it in the corresponding output box. • Once placed in the box, an image of the interior of the output box is acquired for the target position of the next product that goes to this box. 1. If it is detected that the box is full, a warning is sent to the line operator. D1.1 Real world scenarios and metrics for validation definition 64 Step 4: Oiling Hold the toggle with one hand an oil inside the ratchet hole with a brush. Step 5: Assembly, block assembly Place the two small blocks and a spring with oil into one block. Step 6: Assembly, block insertion Press the block (the component that does or does not allow movement of the ratchet) inside the big hole of the wrench. Step 7: Assembly, ratchet insertion Place the ratchet in the big hole of the wrench. Step 8: Assembly, C-shaped ring insertion Place the C-Shaped ring around the ratchet. D1.1 Real world scenarios and metrics for validation definition 65 Step 9: Assembly, C-Shaped ring sealing Seal the C-Shaped ring using pliers in the right hand and a peg in the left hand Step 10: Testing Test the direction of rotation with the Texting fixture. Step 11: Polishing Polish the wrench. Step 12: Package Arrange the wrenches in an orderly fashion. The process can be seen in these videos: • HARTU-INFAR-place_block_ratchet.mp4 • HARTU-INFAR-push-ratchet_place_C-Ring.mp4 6.2.1.2 Inputs Environmental conditions: ▪ Room temperature around 5 to 40oC Personnel: ▪ One assembly operator ▪ One day training for a new operator Components used: D1.1 Real world scenarios and metrics for validation definition 66 ▪ Wrench ▪ C-shaped ring ▪ Block ▪ Ratchet Tools used: ▪ Clamp ▪ Brush ▪ Pliers ▪ Peg ▪ Testing fixture ▪ Polishing cloth Information available: ▪ CAD data for components available 6.2.1.3 Outputs Process output Materials out: ▪ Assembled ratchet wrench 6.2.2 Product specifications HARTU will focus on 3 different assembly tasks: ▪ Step 6 (1-6) • Step 7 (1-7) • Step 8 (1-8) The corresponding parts and tasks are shown in Figure 68. D1.1 Real world scenarios and metrics for validation definition 67 Figure 68. INFAR: Selected operations Figure 69. Parts involved in the INFAR use case D1.1 Real world scenarios and metrics for validation definition 68 The average cycle time for these three operations is less than 30 seconds. 6.2.3 Key figures • Cycle time for the operations: 30 sec. • Number of produced parts / years: 25M 6.2.4 Target state According to some market studies (Figure 70), the hand tool market is growing at a CAGR of 4.2%. To meet the increasing demand for orders, many operators are required to perform the assembly of ratchet wrenches in a dangerous (heavy part manipulation, plating process involving chemicals) and noisy working environment. This causes a labor shortage problem. In addition, the dual digital and green transformation is a critical issue for conventional manufacturing industries like INFAR. HARTU technology and system implementation, can help addressing both labor shortage and dual digital and green transformation. Via AI-enabled multirobotic assembly system developed in HARTU, some operators may be freed from hand tool assembly tasks and can carry out higher-value tasks. Moreover, by adopting robotic automation manufacturing, Carbon emission can be indirectly reduced since automation can improve the efficiency of energy use. 6.3 Demonstrator design 6.3.1 System description • Two robots equipped with specific designed gripper and workpiece loading/unloading mechanism, will automatically assemble a ratchet wrench as shown in the following figure. • A robot cell controller will be included to coordinate the movements of these two robots to complete the assembly task as shown in the following figure where Assembly Planning is mapped with Control Target, Assembly Movement is mapped with two robot commands A Family name 工件名稱 Industrial Scenario Morphology 樣子 Material 材質 Colour 顏色 Brighness 光澤 Weight 重量 Dimensions of the container parallelepiped (mm) 尺寸 x*y*z ratchet wrench (棘輪板手本體) INFAR Parallelepiped Ferromagnetic Any Shiny 100g-500g 320X50X30 C-Shaped Ring (C形環，C扣) INFAR Toroid Ferromagnetic Any Shiny <100g 40X40X1 A component for allowing moving in one direction (擋塊) INFAR Irregular Ferromagnetic Any Shiny <100g 3X3X5 Ratchet (棘輪) INFAR Toroid Ferromagnetic Any Shiny <100g 10X10X10 A component for fixing (固定片) INFAR Irregular Ferromagnetic Any Shiny <100g 12X12X2 Figure 70. Hand tools market D1.1 Real world scenarios and metrics for validation definition 69 & B assigned by the Robot Cell Controller, and there could be some interactions between robots A & B. The Robot Cell Controller can implement the concept illustrated within the purpose dashed block. ➢ Figure 71. Proposed The Robot Cell Controller for multi-robotic assembly Figure 72. From the exploded-view drawing to assembly completeness via robotic assembly process 6.3.2 Components of the system • Two robots • Specific designed gripper for grasping the ratchet and the wrench. • Robot Cell Controller • Centralised robot coordination of the two robots with high-level commands, such as grasping points on the workpieces. • Perception system, consisting of cameras at the ceiling or close to the robot end-effector to provide visual information for further object recognition, grasp planning, etc. • Multi-axis force/torque sensors mounted at the end-effectors to provide contact force information for assembly tasks. • Workpiece loading/unloading mechanism for robot grasping of workpieces for next assembly movement. D1.1 Real world scenarios and metrics for validation definition 70 6.3.3 System deployment The system described above will be first implemented at ITRI located in Hsinchu for further experiments and functionality verification. And the system will be shipped to INFAR located in Changhua (two-hour driving distance from Hsinchu). The core of this system including the robot cell controller, workpiece loading/unloading mechanism, and the software modules for perception functionalities could be delivered to other HARTU Partners for further system testing and development purposes. 6.4 Technical risks CODE Description Probability Impact R.1 The tiny size of workpieces could be the major risk for robot grasping and then assembly. High (4) High (4). Without correct workpiece grasping, the following assembly could not be completed. Very Low=1; Low =2; Medium = 3; High=4; Very High=5 D1.1 Real world scenarios and metrics for validation definition 71 7 ULMA-1: Order preparation: pallet to pallet 7.1 Introduction In logistics, there are different types of order preparation needs depending on how the products arrive at the preparation area and how the orders are delivered. • Input o Products arrive on pallets, this is mainly the case for bulky products, packed in carboards, large cans and sacks. o Small size products arrive in boxes, orderly or randomly distributed. • Output o Products are stacked on pallets, either single or multi-reference. o Products are placed in boxes, sorted or unstacked (randomly). This use case corresponds to the case in which products arrive on mono-reference pallets and are delivered on multi-reference pallets. 7.2 Use case description For the sake of clarity, this use case will be described in reference to the needs of the company BESA S.A., one of ULMA’s customer. In this paint manufacturing company, ULMA has implemented a logistic solution in which products are automatically stacked on pallets at the end of the production line, transported in autonomous AGVs to the warehouse, stored on shelves using an automatic stacker crane and, when required, transported with the Sorting Transfer Vehicles and conveying system to the preparation area, where human operators prepare the orders. 7.2.1 Initial state 7.2.1.1 Process In the order preparation area operators pick units from the incoming pallet (the one that has been transported from the warehouse) and place them on the pallets that will be finally delivered to the customer, as shown in Figure 73. Some of the features of the use case are the following: • The incoming pallets are always monoreference and the output boxes are, usually, multireference. • The warehouse management system informs the operator of the number of units that have to be picked from the incoming pallet. This information is available in a GUI and also is displayed in a pick-to-light system. • Operators manipulate the product by hand, and with the help of industrial manipulators for the heaviest products (they can weight up to 30 kg). D1.1 Real world scenarios and metrics for validation definition 72 • In some few occasions, the incoming pallet transports a box with products inside, which have to be manipulated individually to complete an order (e.g. to take a can from the box an put them on the output pallet). • Operators use their own criteria to create the output pallet, trying to find the best combination to create stable pallets. For that, sometimes they move the already placed items and reposition them. A video of the current process is available here: BESA-Example: Pallet to pallet In the following pictures it is shown the operators manipulating products manually (Figure 74) or with the support of material manipulators (Figure 75). Figure 74. Operator manually placing a box on the pallet Figure 75.Operator placing a can on the pallet with the help of a material handling crane The overall workflow is depicted in the next picture: Figure 73. Real example of output pallet at BESA D1.1 Real world scenarios and metrics for validation definition 73 Figure 76. Pallet to pallet order preparation workflow 7.2.1.2 Inputs ▪ Input pallet with the products to be picked. D1.1 Real world scenarios and metrics for validation definition 80 ▪ Number of items to be picked from the incoming pallet, shown in the pick-to-light system ▪ Input box with the products to be picked. ▪ Empty output box to create the order. ▪ Environmental conditions Category Temp (ºC) Humidity Dry 4-40 <85% Cool 1-5 <95% Frozen -25-1 <85% 8.2.1.3 Outputs ▪ Mono-reference incoming boxes ▪ Multi-reference output boxes ▪ Confirmation of each picking operation through the pick-to-light system 8.2.2 Product specifications ▪ Small boxes, pots, small cans, irregular shape products, blisters, etc. Figure 83. Examples of products manipulated in a box to box scenario 8.2.3 Key figures ▪ Cycle time: from 250 to 1.000 cycles / hour, depending on the product ▪ Error rate (number of items picked or wrong reference):1 / 1.000 pickings 8.2.4 Target state There are many reasons to try to automatize the order preparation operations: • Workforce scarcity: companies are suffering the lack of workforce for this type of job • In some sectors, e.g. pharmacy, companies don’t like human operators to handle products by hand, as they can be medicines with restricted access, products of reduced dimensions and with high economic value. • Operators make mistakes: o grasping the wrong product o wrong number of items. o wrong destination box D1.1 Real world scenarios and metrics for validation definition 81 The scenario envisaged for ULMA is a robotic system, which receives as input information about the product in the box, is able to pick the number of items requested by the WMS and deposits them in the corresponding output box. From an industrial point of view, it is important to achieve a performance as close as possible to that of humans. 8.3 Demonstrator design 8.3.1 System description The objective of this demonstrator is to validate the grasping strategies. The system will be the alternative to current pick-to-light used in logistic centers for manual processes. Products arrive at the picking area inside standard plastic boxes (600 x 400 x 320 mm), that sometimes have internal dividers to create 2 or 4 sub-boxes (see Figure 85). Inside the boxes (or the sub-boxes) only one type of reference is possible. The robot has to pick the products from these boxes (sub-boxes) and deposit them in the output boxes. Figure 84. Flow of products in the scenario: from monoreference to multirefences boxes Figure 85. Standard box (left) and one with one divider to create two sub-boxes 8.3.2 Components of the system To simulate a realistic system we will adapt an already existing robotic cell available at TEK, that allows creating a multi-station setting. Industrial system HARTU demonstrator Input boxes arrive automatically to the picking station. Once products are picked the box is automatically sent to the warehouse. We will have up to 7 input boxes each of them with a monoreference product. We will have one output box D1.1 Real world scenarios and metrics for validation definition 82 Figure 86.Laboratory system layout for order picking in boxes Figure 87. ULMA box-to-box scenario simulation The components in the demonstrator are as follows: • UR 10 mounted on 4m linear axis. Alternatively, the robot can be fixed in one position and the boxes can arrive at the picking station via a conveyor belt. • 2 Finger or suction gripper. o Consider a tool exchanging station or a dual gripper, such as OnRobot Dual Quick Changer - Unchained Robotics D1.1 Real world scenarios and metrics for validation definition 83 Figure 88. Too exchanger components for two tools Figure 89. Dual tool • Up to 7 input box stations (monoreference). • 1 output box station (multireference). • Photoneo L on top of the output box (box status monitoring). o Alternatively, an RGB-D camera on top of the station. • Photoneo L mounted on a linear track above the input box stations. o Alternatively, an RGB-D camera on top of each box. 8.3.3 System deployment The system will be developed and validated in TEKNIKER’s facilities. TEKNIKER will provide all the physical components and integrate them, and the rest of software components developed in HARTU (in collaboration with other partners). 8.4 Technical risks The following risks have been identified in the development of the system: CODE Description Probability Impact R.1 Performance far from that of humans, due to human dexterity and the use of two hands High (4) Medium (3) R.2 Unpredictable product types used in e-commerce. Very High (5) High (4) Very Low=1; Low =2; Medium = 3; High=4; Very High=5 D1.1 Real world scenarios and metrics for validation definition 84 9 Requirements For requirement targets to be assessed by several tests, the number of tests shall be the statistically relevant number of tests. This value shall be decided on a case-by-case basis. The validation methods is sometimes self-evident, and in other cases will be decided prior to experimentation, taking into account the effectiveness of the procedure and possible external constraints. 9.1 User requirements USER REQUIREMENTS Ref. UC Description / target Dem. (Y/N) UR-01 ALL Human operators should work safely in collaboration/presence with robots. Target: Y Y UR-02 ALL An operator shall be able to start and stop the system. Target: Y Y UR-03 ALL A skilled operator should be able to configure a new part handling. Target: Y Y UR-04 ALL An operator should be able to change over the system from one part to another. Target: Y Y UR-05 ALL The users need flexible lines, allowing the use of different box sizes, products and sorting criteria. Target: Y Y UR-06 ALL The operator needs to be informed when the system completes its tasks (e.g., input box empty or output box full) so as to intervene promptly in the line. Target: Y Y UR-07 TCA End user may need to move the sorting system to different places in the plant. Target: Y N UR-08 ALL The operator needs to interact with a user-friendly GUI in order to control the system easily. Target: Y Y The list of user requirements is a preliminary list that will be further expanded in D1.2. 9.2 Functional requirements FUNCTIONAL REQUIREMENTS Ref. UC Description / target Dem. (Y/N) D1.1 Real world scenarios and metrics for validation definition 85 FR-01 TOFAS-1 The robot handles spare parts packaged in carboard boxes. Human operators the rest. Target: Y Y FR-02 TOFAS-1 The robot verifies the reference and number of products manipulated (using the same barcode reading procedure). Target: Y Y FR-03 TOFAS-1 The robot places the objects inside the outbound boxes. Target: Y Y FR-04 TOFAS-1 Both types of output boxes have to be considered: the standard grey big box and carboard smaller boxes. Target: Y Y FR-05 TOFAS-2 Cobot can insert the components. Target: Y FR-06 TOFAS-2 Cobot can screw the bolts. Target: Y FR-07 PCL The system should be able to recognize the parts to be handled . Target: > 95% recognition rate Y FR-08 PCL The system should be able to plan and execute a grasping action. Target: > 95% grasp success Y FR-09 PCL The system should be able to identify empty jig positions. Target: > 95% recognition rate Y FR-10 PCL The system should be able to plan and execute a fixturing action. Target: > 95% fixturing success Y FR-11 PCL The system should be able to plan and execute the opposite removing action. Target: > 95% removing success Y FR-12 PCL The system should be able to pick up and place parts from and onto trays. Target: Y Y FR-13 TCA Detect the size and shape of the product. Target: Y Y FR-14 TCA Pick products from a box (e.g., zucchini). Target: Y Y FR-15 TCA Maximum number of layers of product in a box. Target: <4 Y FR-16 TCA Pick product from a conveyor belt (eggplants and tomatoes). Target: Y N FR-17 TCA Recognize product defects. Target: Y Y FR-18 TCA Place the item neatly in the output box. Target: Y Y FR-19 TCA Do not damage the product during handling operations. Target: Y Y FR-20 INFAR The system shall visually identify the components ready for assembly. Y D1.1 Real world scenarios and metrics for validation definition 86 Target: > 90% recognition FR-21 INFAR The system shall grasp the identified components. Target: > 95% success Y FR-22 INFAR The system shall perform assembly task based on the specified assembly procedures. Target: Y Y 9.3 Non functional requirements NON-FUNCTIONAL REQUIREMENTS Ref. UC Description / target Dem. (Y/N) NR-01 TOFAS-1 Cycle time of the process Target: <60 seconds Y NR-02 TOFAS-1 Collision Detection Rate Target: 100% Y NR-03 TOFAS-1 Successful picking rate Target: (>95% Y NR-04 TOFAS-1 Successful part identification rate. Target: >95% Y NR-05 TOFAS-2 Up time specification. Target: OEE of 95% Y NR-06 TOFAS-2 Cycle time Target: 15 minutes for 10 box kitting preparation Y NR-07 TOFAS-2 Cycle time Target: 90 sec for the assembly operation Y NR-08 PCL Cycle time Target: <10 seconds per part. Y NR-09 PCL Reconfiguration Target: < 15 minutes Y NR-10 PCL System footprint Target: 1 x 1 m Y NR-11 TCA Material compliance with CE Regulation n. 1935/2004 Target: Y N NR-12 TCA Material must be resistant to common washing with water spray from any direction (IP64) Target: Y N NR-13 TCA Cycle time Target: < 3 minutes to complete a box of zucchinis Y NR-14 TCA Cycle time Target: < 3 minutes to complete a box of eggplants Y NR-15 TCA Cycle time Target: < 5 minutes to complete a box of tomatoes Y NR-16 TCA Picking error rate Target: < 1% (1 failed item out of 100 picked) Y D1.1 Real world scenarios and metrics for validation definition 87 NR-17 TCA Compliance with Global G.A.P. Certification Target: Y N NR-18 TCA Classification error Target: according to the tolerances Y NR-19 INFAR Correct assembly on the ratchet wrench passing the testing criteria. Target: Y Y NR-20 INFAR Cycle time Target: < 1 minute for the three selected steps of the ratchet wrench assembly Y NR-21 ULMA-1 Stability of output pallets: Target: Detect misalignments Y NR-22 ULMA-1 Product weight Target: < 30kg Y NR-23 ULMA-1 Stability of output pallet Target: Y Y NR-24 ULMA-1 Type of products. Target: boxes, paint cans, packed bottles Y NR-25 ULMA-1 Cycle time. Although it depends on the product and the final composition of the target pallet, we can consider, as an average: Target: 9 seconds / item Y NR-26 ULMA-1 ULMA-2 Robot shall be able to detect picking and/or placing errors. Target: Y Y NR-27 ULMA-2 Product weight Target: < 5 kg Y NR-28 ULMA-2 Error in picking operations Target: Detect any error Y NR-29 ULMA-2 Cycle time. It depends very much on the product, its arrangement in the box and the gripping mechanism used. For the best case, we can consider: Target: 4 seconds Y NR-30 ULMA-2 Products are place in the destination box randomly but checking that they don’t suffer any damage during dropping. Target: Y Y D1.1 Real world scenarios and metrics for validation definition 88 10 Preliminary Risk assessment A Preliminary risk assessment has been done based on the analysis of the use cases for an early detection of possible risks. This risk assessment is to be updated per use case regularly, including the mitigation measures. Risk-analysis No Use Case Occurrence User phase Cause Effect Remarks Seriousness Exposure Probability Danger avert Risk level Class 1 All Robot moves to pick position All Moving part approaches Static part Getting stuck Use of cobots at cobot requirements (e.g. speed) 1 2 1 1 3 1: Low (possibly acceptable) 2 All Robot picks product from pick positions Automatic Sharp parts Cutting Use of cobots at cobot requirements (e.g. speed) 1 2 1 2 4 1: Low (possibly acceptable) 3 All Gripper picks product All Moving part approaches static part Getting stuck, Cutting 1 3 3 1 5 2: Middle (improvement necessary) 4 All Robot movement All Moving part approaches static part Being hit by moving part Use of cobots at cobot requirements (e.g. speed) 1 3 3 1 5 2: Middle (improvement necessary) 5 All Robot moves to place positions All Moving part approaches Getting stuck Use of cobots at cobot requirements (e.g. speed) 1 2 1 1 3 1: Low (possibly acceptable) 6 All Robot places product Automatic Sharp parts Cutting Use of cobots at cobot requirements (e.g. speed) 1 1 2 2 4 1: Low (possibly acceptable) 7 All Robot moves from Maintenance, service, Manual Sharp parts Cutting After maintenance the robot can be in undefined position 1 2 2 1 5 2: Middle (improvement necessary) D1.1 Real world scenarios and metrics for validation definition 89 undefined positon 8 All Working with electronics Maintenance, installation Live parts Electrocution 2 1 1 2 4 1: Low (possibly acceptable) 9 All Air hose lets loose All Bad installation Being hit by moving part 1 1 1 1 1 1: Low (possibly acceptable) 10 All User can get into the blind spot of safety measures All Safety scanners cannot scan all area Getting stuck, being hit 1 2 1 2 4 1: Low (possibly acceptable) 11 All User can make unsafe robot program All Flexibility is target of systems Getting stuck, being hit Damage to machine 1 1 2 2 4 1: Low (possibly acceptable) 12 TOFAS Mobile manipulator moves All Moving manipulator Getting stuck, being hit 1 1 2 2 4 1: Low (possibly acceptable) 13 ULMA_1 Product drops All Gripping force not enough Emergency shutdown, power down Getting stuck, being hit 2 1 1 3 4 1: Low (possibly acceptable) 14 INFAR Robots move towards each other All Moving part approaches moving part Getting stuck 1 3 3 1 5 2: Middle (improvement necessary) 16 PCL Movement of jig All Moving part approaches Static part Getting stuck 1 2 1 1 3 1: Low (possibly acceptable) 17 TCA All Moving part approaches Getting stuck, being hit 1 1 1 1 1