Robotic Shelf Replenishment by Combining Non-Prehensile Object Manipulation with Simple Grasping

Robotic Shelf Replenishment by Combining Non-Prehensile Object Manipulation with Simple Grasping Leonidas Koutras, Sotiris Stavridis, Christos Papakonstantinou and Zoe Doulgeri Abstract— In this work, the problem of robotic shelf replenishment is being studied. Such tasks involve a variety of object types and geometries, which should be picked from boxes where they are tightly packed and placed on shelves in tight formations with the appropriate orientation. We consider a bimanual robotic set up with a parallel finger gripper and a 3Dprinted end-effector and propose to combine simple grasping with a set of non-prehensile manipulations to achieve such a replenishment task. This work reports on the details of the implementation of the proposed strategy and on the initial investigation of the feasibility and effectiveness of the proposed solution for a representative set of super-market products, demonstrated in a number of experiments in the lab. I. INTRODUCTION One of the most crucial functions in retail environments is the shelf replenishment process, directly connecting the consumer with the product. The automation of such processes enhances the productivity of the business, while reducing costs and labor intensive non-ergonomic human activities [1]. A significant step in the direction of replenishment automation is the introduction of robotic systems in retail stores [2], which have seen growing acceptance among customers and owners in recent years [3], [4]. The problem of robotic shelf replenishment is challenging since the robot needs to be able to manipulate a wide variety of objects. The objects are usually tightly packed in boxes, and need to be appropriately placed in confined spaces in shelves [5] in tight formations with their label visible to the customer. Therefore, to successfully execute the complete task, multiple technologies are required to be combined, like object recognition and tracking [6]–[8], grasp planning and execution [9]–[11], control [12], [13], decision making [14], [15] as well as tools for optimizing the object selection and placing position on the shelf [16], [17]. A major differentiating factor between approaches in the literature is the utilized end-effector. Suction grippers [18], [19], parallel grippers [5], [20], [21] or more elaborate gripper designs [22]–[24] have been utilized. While suction grippers are very efficient and simple in grasping specific types of items, they fail to generalize to a wide variety of object materials and geometries. Furthermore, cluttered environments can also be problematic, as a suction gripper offers limited affordances. On the other hand, complex Authors are with the Automation & Robotics Lab, Dept. of Electrical & Computer Engineering, Aristotle University of Thessaloniki, Greece {kleonidas, sotistav, papachri, doulgeri}@ece.auth.gr This work has received funding from the European Union’s Horizon Framework Programme for Research and Innovation under grant agreement no 101120823, project MANiBOT. gripper designs introduce complicated control architectures and require sophisticated algorithms. Parallel grippers offer a middle ground between the two approaches, where increased dexterity, compared to suction grippers, is combined with simplicity. With the utilization of external dexterity, such grippers can also use pivoting to change the in-hand orientation of the grasped object [12], [25]. However, parallel grippers may fail to grasp objects in cluttered environments while in-hand manipulation methods are prone to errors requiring additional sensorial feedback to increase accuracy. The utilization of mobile or bimanual robots has also been shown to enhance the performance of shelf replenishing robots. Mobile robots [9], [10], [26] are able to autonomously navigate within stores in the presence of humans [27] to reach various shelves, and are also able to expand their local workspace for replenishment tasks. Bimanual robots offer increased degrees of freedom and larger flexibility in the execution of the desired task, both increasing the efficiency of the robot, and offering additional capabilities [19], [28], [29]. Such bimanual systems have also been combined with teleoperation setups for replenishment tasks [30]. While multiple methods have been proposed for this task, the utilization of non-prehensile manipulation primitives to enhance the capabilities of a replenishing robot is relatively unexplored. The majority of developed approaches follow a recognition - grasping - pick and place strategy, accompanied with task planning algorithms [31]. Such methods, however, often struggle in cluttered spaces with limited grasp affordances, or require specialized hardware and complicated grasp planning and control architectures. Non-prehensile manipulation primitives on the other hand, can be used to either create grasp affordances in clutter or to achieve tight placement configuration of objects with decreased control and hardware complexity [32]. Non-prehensile manipulation methods have been proposed to push [33], [34] or drag [35], [36] small objects by single manipulators, or for sizable objects bimanual [37] and full body manipulation [38] approaches have been used. The combination of non-prehensile manipulations with grasping has also been proven to be successful, enabling the robot to grasp objects by effectively manipulating clutter [21], [39]–[41]. The combination of non-prehensile tasks with grasping in retail has been studied in [24] in depalletizing tasks. There, the authors utilize a bimanual robot, involving a small conveyor belt end-effector and a custom anthropomorphic gripper, imitating human-like object grasping. While their method has a high success rate, the system has increased hardware and control complexity for both the gripper and Fig. 1: Products used in this paper (a) Biscuit box (b) Gyros box (c) Soft biscuit box (d) Shampoo bottle (e) Vinegar bottle (f) Shaving foam (g) Coffee bag Fig. 2: Tightly packed initial configurations of retail items the conveyor belt, as well as the necessity of specialized custom hardware. In this paper, we propose a shelf replenishment bimanual strategy where a library of non-prehensile manipulation primitives is combined with simple grasping actions. Our method is hardware independent utilizing an of-the-shelf parallel gripper performing simple grasping actions without requiring complex grasp planning algorithms and a small 3Dprinted end-effector enabling multiple manipulation primitives. By appropriately sequencing those primitives, we show that the replenishment task can be successfully completed for a wide variety of retail objects with different shapes, materials, deformability and size. The objects are manipulated from a densely cluttered configuration and are tightly placed in a shelf, imitating real world retail environments. II. MOTIVATION The replenishing task requires the picking of items from a tightly packed configuration and their tight placement on a shelf that is almost empty with the appropriate orientation so that the product label is visible to the customer. Such items are of different geometry, size and deformability. A representative set considered in this work is shown in Figure 1. The majority of these products in tightly packed configurations offer limited grasping affordances to a parallel gripper. It is clear for example that grasping of an item from the tightly packed configuration shown in Figure 2 with a simple parallel gripper may not be possible. On the other hand, a suction gripper could also fail due to the shape and flexibility of some items like the coffee bags. The tight placement of these items on a shelf may also be jeopardized by the gripper geometry that can induce collisions with the environment and the rest of the items on the shelf. Moreover, a confined and limited space does not allow elaborate decluttering actions by pushing items away. For an efficient task execution, it would also be preferable to grasp the item in a way that allows quick and feasible placement in the shelf thus avoiding re-grasping actions. For example, grasping an item from the top (assumed feasible) may prevent its direct placement due to the restricted space of the shelf that prevents top reaching of the gripper / object on the shelf surface. Furthermore, the confined space of the shelf as well as the existence of adjacent products limit the collision-free configurations of the robot for reaching the desired final position of the grasped object. In fact, objectside grasps would avoid such collision problems as most of the shelves have a height that is slightly bigger than that of the items. However, the initial tight configuration may prevent object-side grasps in many item cases. Moreover, in shelf replenishment objects need to be tightly packed in the shelf. In side grasps, gripper fingers may prevent products to be directly placed in contact with each-other. III. PROPOSED METHOD To address the challenges explained in Section II we consider a bimanual robotic system operating in front of a retail shelf that is required to be replenished. The bimanual robot is equipped with a simple parallel gripper and a 3Dprinted end-effector for efficient object manipulation. The robot is also equipped with a perception system in order to detect the objects to be manipulated. We assume that the product box is placed at an appropriate position in front of the shelf and it is opened from two adjacent sides before task execution to enable object manipulation. The box opening is considered beyond the scope of this work; it could be done by a human employee or even by the robot with additional custom hardware on its end-effectors. We propose the utilization of non-prehensile object manipulations combined with simple grasping and placing tasks. Non-prehensile object manipulations aim at creating side grasp affordances for picking the tightly packed items from the box and ensuring the tight formation of the objects on the shelf with the correct orientation so that the label is visible to the customers. Thus, object side grasping and front placing on the shelf is performed from / to an uncluttered space, and re-grasping is avoided. Non-prehensile manipulation primitives include tumbling, dragging, pushing and rotating. Tumbling, dragging and rotating are used to create side grasp affordances of items in tight configurations. See, for example, the case of Figures 2a, 2b and 2c respectively. There are of course cases where the geometry of the item leaves free space between adjacent objects allowing a direct side grasp; for example with vinegar bottles in Figure 2e, the shampoo bottle in Figure 2d or the shaving foam case in Figure 2f. Top grasps are also available for flexible products like the coffee bag in Figure 2g. Pushing is used to achieve the tight formation on the shelf after placement. Linear pushing is usually adequate (Figures 3a, 3b) while in some cases by appropriately selecting the initial (a) Side linear push - flat end-effector (b) Front linear push - gripper (c) Rotating push - gripper Fig. 3: Object pushing instances (a) Tumbling biscuit box (b) Rotating biscuit box (c) Grasping coffee bag (d) Placing coffee bag Fig. 4: Manipulation exploiting external dexterity. relative object-pusher position, the object is also rotated while being pushed to reach a desired orientation (Figure 3c). The latter is needed in products for which the gripper aperture is smaller than the front item side and hence they are grasped from their narrower side like for example the gyros box (Figure 6b); thus after their placement should be rotated to the required orientation. In most cases, the gripper is performing grasping and placing actions and the 3D printed end-effector the nonprehensile manipulations. However, for faster execution, when pushing from the front is needed after placing the object, the gripper can execute such simple non-prehensile manipulation actions (Figures 3b, 3c). The area contact achieved by the 3D-printed tool ensures stable and accurate pushing. Pivoting like in the tumbling case is performed by the edge of the 3D-printed tool i.e. with a line contact that may be prone to errors. However, adjacent items can provide external dexterity to achieve accuracy in such tasks (Figure 4a). This is also the case for rotating an item using its surrounding items in Figure 4b. External dexterity from gravity is also used in deformable items such as coffee bags. They can be grasped from the top rim and remain upright due to gravity and deformability when the gripper bends to perform front placement on the shelf (Figures 4c, 4d). In this work, the sequence of primitives is predefined for each product in the set of objects considered. Such high level decision making can be automated using either carefully designed heuristics or learning based approaches. For the biscuit boxes, shown in the initial configuration (a) Tumbling (b) Front grasping (c) Front placing (d) Front pushing (e) Side Pushing Fig. 5: Primitive sequence for the biscuit box (a) Dragging (b) Side grasping (c) Side placing (d) Rotating pushing (e) Side Pushing Fig. 6: Primitive sequence for the gyros box (a) Rotating (b) Front grasping (c) Front placing (d) Front pushing (e) Side Pushing Fig. 7: Primitive sequence for the soft biscuit box (a) Front grasping (b) Front placing (c) Front Pushing (d) Side pushing Fig. 8: Primitive sequence for the vinegar bottle (a) Diagonal grasp (b) Front grasp (c) Front placing (d) Front pushing (e) Side Pushing Fig. 9: Primitive sequence for the shaving foam (a) Front pre-grasp (b) Slide down (c) Front grasp (d) Front placing (e) Front pushing (f) Side Pushing Fig. 10: Primitive sequence for the shampoo bottle in Figure 2a, the following sequence is used: tumbling (Figure 4a) →grasp (Figure 5b) →front placing (Figure 5c) →front pushing (Figure 5d) →side pushing (Figure 5e). For the gyros boxes which are larger than the maximum gripper aperture and are shown in the initial configuration in Figure 2b the following sequence is used: dragging (Figure 6a) →side grasping (Figure 6b) →side placing (Figure 6c) →rotating push (Figure 6d) →side pushing (Figure 6e). The biscuit boxes with triangular top shown in Figure 2c do not allow tumbling due to their shape. Thus, the sequence used is: rotating (Figure 7a) →front grasping (Figure 7b) →front placing (Figure 7c) →front push (Figure 7d) →side pushing (Figure 7e). For objects with grasping affordances, there is (a) Pinching above (b) Front placing (c) Front Pushing (d) Side pushing Fig. 11: Primitive sequence for the coffee bag no need to manipulate them before grasping them. Such objects are the vinegar bottle whose initial configuration is shown in Figure 2e, with the sequence: front grasping at mid height (Figure 8a) →front placing (Figure 8b) →front pushing (Figure 8c) →side pushing (Figure 8d) and the shaving foam with initial configuration shown in Figure 2f and sequence front/diagonal grasping (Figures 9a, 9b) →front placing (Figure 9c) →front pushing (Figure 9d) →side pushing (Figure 9e). For the case of the shaving foam, front or diagonal grasps are selected depending on the position of each item in the initial clutter. Notice that diagonal grasps allow for front placing. An interesting case is the shampoo bottle whose initial configuration is shown in Figure 2d, which may have available grasp affordances near the top, but such affordances do not generate stable grasps. Thus, a different strategy is followed: front pre-grasp (Figure 10a) →slide down to declutter (Figure 10b) →grasp (Figure 10c) →front placing (Figure 10d) →front pushing (Figure 10e) →side pushing (Figure 10f). Finally, for the coffee bags with the initial configuration shown in Figure 2g the flexibility of the object is used for the sequence: pinch the rim from above (Figure 11a) →front placing (Figure 11b) →front pushing (Figure 11c) →side pushing (Figure 11d). IV. IMPLEMENTATION The process of the entire shelf replenishment task is explained in Algorithm 1. The item selection and placement on the shelf is executed sequentially from left to right for each row of products. Notice that since the shelf and product geometries are known, it is trivial to extract a grid of the desired final product positions on the shelf. Further notice that each sequence has one grasping action and one placing action, accompanied by non-prehensile manipulation actions. For the first item in each row, the sequence does not include the final side pushing action, since there is enough space for the item to be placed in the correct horizontal position. A. Inverse kinematics and Bimanual Synchronization under constraints. To address the challenge of coordinating the two arms of the bimanual robot, while avoiding a strict scheduler between the tasks of the two arms, a hierarchical optimization methodology [42] is employed to generate joint velocities that satisfy safety constraints and objectives at each control cycle, as described in [43]. In particular, we have adopted the following hierarchy of objectives and constraints: cjl ≻ccol ≻cint ≻cfr (1) where cjl, refers to joint position and velocity limits, ccol to collision avoidance that includes self and obstacle collision avoidance. It is clear that the highest priority constraints correspond to the hardware limits of the robot and the safety constraints. The task objectives are separated into two priority levels, and task prioritization is dynamically assigned based on their criticality, with cint referring to object interaction tasks and cfr to free-space motion. The arm that interacts with objects performing non-prehensile Algorithm 1: Shelf Replenishment Input : Product id, manipulation action sequence for the product Extract item placing grid on shelf while shelf has free space and box is not empty do Select an item from the product box Detect item pose in the product box while item is not grasped do Select next action from sequence Move to the initial contact point(s) while termination condition is not met do Execute the manipulation controller end end Select next item final position from the shelf grid Detect free space around the final position Place the item on free space centroid while manipulation sequence is not finished do Select next action from sequence Move to the initial contact point(s) while termination condition is not met do Execute the manipulation controller end end end manipulations or grasping, where precise motion is necessary for the success of the task, receives higher priority than the arm that performs free-space motion. Thus, when, for example, the free-space motion interferes with the arm performing a grasping task, motion adjustments for selfcollision avoidance are performed only by the arm executing the lower-priority free-space motion. Assume that the free-space moving arm moves between two poses following the reference trajectory pd(t),Qd(t). Let the tracking error be defined as: e=p−pd 2 log (Q∗Qd)(2) with p∈R3,Q∈ S3the robot position and orientation as a unit quaternion, respectively, and log(·)the quaternion logarithm [44]. Since the arm performing free-space motion has lower priority, it will often need to avoid the arm performing object interaction tasks. In that case, we implement a phase stopping mechanism to stop the evolution of the reference trajectory in order to make the free-space moving robot wait for the completion of the interaction task. Notice that if this mechanism were not implemented, the reference trajectory would continue its evolution, leading to large tracking errors, causing abrupt movements when motion restarts. To pause the reference trajectory evolution, instead of providing pd(t), Qd(t)to the robot, we provide at each time step pd(g(t)), Qd(g(t)), with g(t)chosen so that: ˙g(t) = (1,∥e∥ ≤ ϵ 1−f(∥e∥ − ϵ),∥e∥> ϵ (3) p1p2 p1p2 p1p2 (a) Front / Side push p1p2 p1p2 p1p2 (b) Rotating push Fig. 12: Pushing primitive - top view with ϵ > 0a small constant and f(∥e∥ − ϵ)a function that guarantees a smooth transition from 1to 0. Notice that when the free-space motion constraint is satisfied, the errors are small with ∥e∥ ≤ ϵand ˙g(t)=1. Thus, the reference trajectory is being executed normally. On the other hand, when the constraints are violated and large errors occur, ˙g(t)=0halting the trajectory evolution. B. Non-Prehensile Manipulation Primitives 1) Pushing Primitive: This primitive requires free space around the object side opposite to the object goal, for the endeffector to push it. The end-effector is initialized in contact with the object on this side, and moves on a straight line towards the desired direction using position control. The pushing action is completed when either a goal position is reached, for objects in free space, i.e. empty shelves, or until a collision with another object is detected by measuring the end-effector force. The area contact of the 3D-printed end-effector, or the two finger contact of the gripper - equivalent to a line contact, increase the robustness of the approach. We consider two cases of pushing. By initializing the line contact or the contact area parallel to the object surface, the robot reduces unwanted rotations during pushing, successfully moving the object at a straight line as shown in Figure 12a. This method is used for front and side pushing. On the other hand, when the end-effector line or area is not parallel with the object, the robot rotates the object while pushing, which is used to achieve both a desired position and a desired orientation as shown in Figure 12b. 2) Tumbling Primitive: This manipulation action requires free space above the object, manipulation affordances on the top of the object and enough space for it to fall in order to grasp it afterwards. The end-effector is initialized in contact with the top surface of the object and applies forces to rotate the object around a pivot point ppas shown in Figure 13. The contact point pis selected close to the boundary of the top side to allow better torque application at pp. The control objective is to rotate the object so that for the rotation angle θreaches a desired value θd, without sliding on the object surface. To achieve this, we utilize an admittance controller: M¨ p+D˙ p=f+Rfd(4) where M,D∈R3×3are diagonal positive definite matrices, p,f∈R3are the end-effector position and measured force, x z {W}pp x′ z′ {O} po θ p l d w Fig. 13: Tumbling primitive - side view. R∈ SO(3) is the rotation matrix corresponding to the object orientation and the desired force fd∈R3is given by: fd=  κtp(θ−θd) + κtd ˙ θ+κti Rt 0(θ−θd)dξ 0 −fn (5) where κtp, κtp, κtp, fn>0are design parameters. Notice that object rotation is achieved by the first row of (5) by applying appropriate tangential forces to the object side. The last row of (5) is responsible for keeping the forces within the friction cone. By selecting a sufficiently large fnthe robot does not slide on the surface of the object, achieving the desired objective. Notice that the torque applied to pp about the yaxis of the world frame is given by: τy=lfx−dfn+mg∥r∥sin ϕ(6) where lis the object length, fxthe first row of (5) and d the distance between pand the object boundary, mis the object mass, gis the gravitational constant, r=pp−po, the displacement between the object center of mass and pp, and ϕis the angle between rand the direction of the gravity force. In order to mitigate the negative effect of fn,pis selected as close to the boundary as possible, according to the geometry of the end-effector. In case the object center of mass aligns with the object geometric center - which is a common occurrence in retail items - (6) simplifies to: τy=lfx−dfn+1 2mgpl2+w2sin θ−tan−1w l (7) with wthe object width. Notice that the torque related to gravity hinders the tumbling motion for θ < tan−1w l and assists it for θ > tan−1w l. Thus, setting θd> tan−1w lone can ensure successful tumbling. Larger θd offers robustness to uneven mass distributions and larger d. For fragile objects that are not allowed to fall, this primitive can be executed in a bimanual manner, where the desired x′ z′ {O} popod p Fig. 14: Dragging primitive - side view angle θdis selected small enough so that the object does not fall and the second manipulator moves to grasp the tilted object before retracting the arm that performs the tumbling. 3) Dragging Primitive: The dragging primitive is selected for tightly packed objects, horizontally stacked on top of each other as in Figure 14. The only requirement for this primitive is that there is enough space on the top surface of the object to effectively manipulate it. The object frame {O}is selected so that its x′axis is aligned with the dragging direction. The initial contact point pselection in this case is arbitrary. The control objective is to move the object to the desired relative position pod chosen so that the object can be subsequently side grasped, without sliding on the surface of the object. This is achieved by utilizing again an admittance control setup (4) with the desired force given by: fd=  −κdp(po−pod)−κdd ˙po−κdi Rt 0(po−pod)dξ 0 −fn (8) where κdp, κdd, κdi >0. The sliding motion is produced by the first row of (8) while fnis again responsible for maintaining the contact force within the friction cone. 4) Rotating Primitive: The rotating primitive is similar to the tumbling primitive, but instead of applying forces on the top of the object, the end-effector applies forces on the object side. To stabilize the pivot point, the robot either utilizes environmental affordances, by taking advantage of other objects in its surroundings, or the second end-effector is used. This case is shown in Figure 15 where two objects are in contact. The robot applies forces at pwhile the second end-effector creates a supporting area contact shown in cyan between points p1and p2. Thus, the object rotates about the pivot point ppto reach a desired angle θdso that a front or side grasp is possible. The control law is again similar to the other cases, with an admittance controller given by (4) with fdsimilar to (5) given by: fd=  −fn κrp(θ−θd) + κrd ˙ θ+κri Rt 0(θ−θd)dξ 0 (9) with κrp, κrd, κri >0. Notice that in this case the torque applied to ppdoes not include any gravitational terms, thus one only has to take into consideration the contact point distance from the object side boundary. V. EXPERIMENTAL RESULTS To test the designed method, a bimanual robot with two UR5e robotic manipulators is used. The manipulators are x y {W} ppx′ y′ {O} θ p1 p2 p Fig. 15: Rotating primitive - top view (a) Biscuits (b) Gyros Box (c) Soft biscuits (d) Shampoo bottle (e) Vinegar bottle (f) Shaving foam (g) Coffee bag Fig. 16: Final configurations of retail items on the shelf equipped with two ATI mini40 force torque sensors to measure the forces applied to the end-effectors. A Robotiq 2F-140 gripper is used for grasping and a 3D-printed endeffector is placed on the second arm. To obtain visual feedback, an Intel RealSence D415 RGB-D sensor is placed above the robot workspace. For visual object detection and 3D pose estimation of the products, the Segment Anything Model by Meta AI Research [45] is utilized along with the corresponding depth information in a semi-automated approach with minimal user input. We conducted shelf replenishment experiments for all items in Figure 1, where an empty shelf was replenished by the robot, using four to six items of each category, with initial positions shown in Figure 2. The robot was successful in completing the replenishment process for all available products, completely replenishing the shelf, with all products correctly placed with their final configurations on the shelf shown in Figure 16. Instances of a manipulation sequence for each product are shown in Figures 5 - 11 with more details in the supplementary video1. A. Discussion and future work The experimental analysis in this paper operates as a proof of concept of the effectiveness of the proposed solution of combining the intuitiveness of a simple parallel gripper 1The experiments can also be found in the following link: https:// www.youtube.com/watch?v=VzOqW8LJ4bM with non-prehensile manipulation allowing for objects to be grasped from dense clutter as well as to be placed in tight formations. However, this approach faces limitations owing to the requirement to design a manipulation primitive sequence for each product by hand, as well as the limited feedback given by the semi-automated object sensing and the lack of robot mobility. Improvements in those areas by using a mobile robot, equipped with tools to automate primitive sequence extraction and real-time object tracking, would greatly enhance the robustness of the proposed method. VI. CONCLUSIONS In this paper a shelf replenishment strategy using a sequence of non-prehensile manipulation and grasping was studied. The utilization of non-prehensile manipulation is successful in creating grasp affordances for tightly packed items, as well as placing them in cluttered, confined places in shelves. Thus, the robot can manipulate a large variety of retail objects with a parallel gripper, overcoming the limitations of suction grippers or the complexities of complicated gripper designs. Experimental results showcase the proposed solution effectiveness for a wide variety of retail objects, with different materials, sizes, shapes and flexibility. REFERENCES [1] G. B¨ uy¨ uk¨ ozkan and F. G¨ oc¸er, “Digital supply chain: Literature review and a proposed framework for future research,” Computers in Industry, vol. 97, pp. 157–177, 2018. [2] R. Bogue, “Strong prospects for robots in retail,” Industrial Robot: the international journal of robotics research and application, vol. 46, pp. 326–331, 2019. [3] L. Lu, R. Cai, and D. Gursoy, “Developing and validating a service robot integration willingness scale,” International Journal of Hospitality Management, vol. 80, pp. 36–51, 2019. [4] Y. Wang, Q. Kang, S. Zhou, Y. Dong, and J. Liu, “The impact of service robots in retail: Exploring the effect of novelty priming on consumer behavior,” Journal of Retailing and Consumer Services, vol. 68, p. 103002, 2022. [5] M. Costanzo, G. De Maria, G. Lettera, and C. Natale, “Can robots refill a supermarket shelf?: Motion planning and grasp control,” IEEE Robotics & Automation Magazine, vol. 28, no. 2, pp. 61–73, 2021. [6] G. A. G. Ricardez, T. Wakayama, S. Ikemura, E. Fujiura, P. M. U. Eljuri, H. Ikeuchi, M. Yamamoto, L. E. Hafi, and T. Taniguchi, “Toward resilient manipulation of food products: Analysis of 6dpose estimation at the future convenience store challenge 2022,” in 2023 IEEE 19th International Conference on Automation Science and Engineering (CASE), 2023, pp. 1–6. [7] A. Savit and A. Damor, “Revolutionizing retail stores with computer vision and edge ai: A novel shelf management system,” in 2023 2nd International Conference on Applied Artificial Intelligence and Computing (ICAAIC), 2023, pp. 69–74. [8] C. Rauch, R. Long, V. Ivan, and S. Vijayakumar, “Sparse-dense motion modelling and tracking for manipulation without prior object models,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 11 394– 11 401, 2022. [9] B. Max, B. James, C. Richard, H. Dan, K. Lukas, K. Dan, L. John, M. Jeremy, M. Carolyn, M. Frank, P. Chavdar, P. Josh, S. Krishna, and T. Mark, “Revolutionizing retail stores with computer vision and edge ai: A novel shelf management system,” in Proceedings of Robotics: Science and Systems, 2023. [10] G. A. G. Ricardez, S. Okada, N. Koganti, A. Yasuda, P. M. U. Eljuri, T. Sano, P.-C. Yang, L. E. Hafi, M. Yamamoto, J. Takamatsu, and T. O. and, “Restock and straightening system for retail automation using compliant and mobile manipulation,” Advanced Robotics, vol. 34, no. 3-4, pp. 235–249, 2020. [11] A. Cavallo, M. Costanzo, G. De Maria, C. Natale, S. Pirozzi, S. Stelter, G. Kazhoyan, S. Koralewski, and M. Beetz, Robotic Clerks: Autonomous Shelf Refilling. Cham: Springer International Publishing, 2022, pp. 137–170. [12] M. Costanzo, S. Stelter, C. Natale, S. Pirozzi, G. Bartels, A. Maldonado, and M. Beetz, “Manipulation planning and control for shelf replenishment,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 1595–1601, 2020. [13] J. Z. Woodruff and K. M. Lynch, “Planning and control for dynamic, nonprehensile, and hybrid manipulation tasks,” in 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017, pp. 4066–4073. [14] D. Roy, S. Nigam, R. de Koster, I. Adan, and J. Resing, “Robotstorage zone assignment strategies in mobile fulfillment systems,” Transportation Research Part E: Logistics and Transportation Review, vol. 122, pp. 119–142, 2019. [15] M. Merschformann, T. Lamballais, M. de Koster, and L. Suhl, “Decision rules for robotic mobile fulfillment systems,” Operations Research Perspectives, vol. 6, p. 100128, 2019. [16] Y. S. Liang, D. Pellier, H. Fiorino, S. Pesty, and M. Cakmak, “Simultaneous end-user programming of goals and actions for robotic shelf organization,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018, pp. 6566–6573. [17] N. Abdo, C. Stachniss, L. Spinello, and W. Burgard, “Robot, organize my shelves! tidying up objects by predicting user preferences,” in 2015 IEEE International Conference on Robotics and Automation (ICRA), 2015, pp. 1557–1564. [18] G. A. G. Ricardez, P. M. U. Eljuri, Y. Kamemura, S. Yokota, N. Kugou, Y. Asama, Z. Wang, H. Kumamoto, K. Yoshimoto, W. Y. Chan, T. Nagatani, P. Tulathum, B. Usawalertkamol, L. E. Hafi, H. Ikeuchi, M. Yamamoto, J. Takamatsu, T. Taniguchi, and T. O. and, “Autonomous service robot for human-aware restock, straightening and disposal tasks in retail automation,” Advanced Robotics, vol. 36, no. 17-18, pp. 936–950, 2022. [19] R. Sakai, S. Katsumata, T. Miki, T. Yano, W. Wei, Y. Okadome, N. Chihara, N. Kimura, Y. Nakai, I. Matsuo, and T. S. and, “A mobile dual-arm manipulation robot system for stocking and disposing of items in a convenience store by using universal vacuum grippers for grasping items,” Advanced Robotics, vol. 34, no. 3-4, pp. 219–234, 2020. [20] T. Motoda, D. Petit, T. Nishi, K. Nagata, W. Wan, and K. Harada, “Shelf replenishment based on object arrangement detection and collapse prediction for bimanual manipulation,” Robotics, vol. 11, no. 5, 2022. [21] M. Kiatos, L. Koutras, I. Sarantopoulos, and Z. Doulgeri, “Learning a pre-grasp manipulation policy to effectively retrieve a target in dense clutter,” in 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024, pp. 7543–7549. [22] N. George, S. Saha, S. Parab, V. Vakharia, R. Lima, V. Vatsal, and K. Das, “System for autonomous management of retail shelves using an omnidirectional dual-arm robot with a novel soft gripper,” in 2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2024, pp. 2631–2637. [23] A. Causo, Z.-H. Chong, R. Luxman, Y. Y. Kok, Z. Yi, W.-C. Pang, R. Meixuan, Y. S. Teoh, W. Jing, H. S. Tju, and I.-M. Chen, “A robust robot design for item picking,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018, pp. 7421–7426. [24] M. Garabini, D. Caporale, V. Tincani, A. Palleschi, C. Gabellieri, M. Gugliotta, A. Settimi, M. G. Catalano, G. Grioli, and L. Pallottino, “Wrapp-up: A dual-arm robot for intralogistics,” IEEE Robotics & Automation Magazine, vol. 28, no. 3, pp. 50–66, 2021. [25] F. E. Vi˜ na B., Y. Karayiannidis, C. Smith, and D. Kragic, “Adaptive control for pivoting with visual and tactile feedback,” in 2016 IEEE International Conference on Robotics and Automation (ICRA), 2016, pp. 399–406. [26] R. Kai, K. Ohashi, H. Fujita, T. Kojima, Y. Sasaki, M. Niitsuma, and K. Umeda, “Development of an environmentally independent mobile manipulation system for product disposal in retail stores,” in 2024 IEEE/SICE International Symposium on System Integration (SII), 2024, pp. 1282–1287. [27] L. Liu, D. Dugas, G. Cesari, R. Siegwart, and R. Dub´ e, “Robot navigation in crowded environments using deep reinforcement learning,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020, pp. 5671–5677. [28] Y.-S. Su, S.-H. Lu, P.-S. Ser, W.-T. Hsu, W.-C. Lai, B. Xie, H.-M. Huang, T.-Y. Lee, H.-W. Chen, L.-F. Yu, and H.-C. Wang, “Poseaware placement of objects with semantic labels - brandname-based affordance prediction and cooperative dual-arm active manipulation,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019, pp. 4760–4767. [29] C. Smith, Y. Karayiannidis, L. Nalpantidis, X. Gratal, P. Qi, D. V. Dimarogonas, and D. Kragic, “Dual arm manipulation—a survey,” Robotics and Autonomous Systems, vol. 60, no. 10, pp. 1340–1353, 2012. [30] R. Lima, S. Saha, N. George, V. Vakharia, S. Parab, S. Gaonkar, V. Vatsal, and K. Das, “Teleoperated omni-directional dual arm mobile manipulation robotic system with shared control for retail store,” in 2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2024, pp. 2935–2942. [31] C. H. Corbato, M. Bharatheesha, J. van Egmond, J. Ju, and M. Wisse, “Integrating different levels of automation: Lessons from winning the amazon robotics challenge 2016,” IEEE Transactions on Industrial Informatics, vol. 14, no. 11, pp. 4916–4926, 2018. [32] F. R. Hogan, J. Ballester, S. Dong, and A. Rodriguez, “Tactile dexterity: Manipulation primitives with tactile feedback,” in 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020, pp. 8863–8869. [33] J. Lloyd and N. F. Lepora, “Goal-driven robotic pushing using tactile and proprioceptive feedback,” IEEE Transactions on Robotics, vol. 38, no. 2, pp. 1201–1212, 2022. [34] J. St¨ uber, C. Zito, and R. Stolkin, “Let’s push things forward: A survey on robot pushing,” Frontiers in Robotics and AI, vol. 7, 2020. [35] S. Sampaziotis and Z. Doulgeri, “A model free robot control method for dragging an object on a planar surface by applying top contact forces,” in 2022 International Conference on Robotics and Automation (ICRA), 2022, pp. 8361–8367. [36] M. M. G. Ardakani, J. Bimbo, and D. Prattichizzo, “Quasi-static analysis of planar sliding using friction patches,” The International Journal of Robotics Research, vol. 39, no. 14, pp. 1775–1795, 2020. [37] L. Koutras, I. Ntoliou, and Z. Doulgeri, “Towards passivity based nonprehensile bimanual manipulation of large objects,” in 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids), 2023, pp. 1–8. [38] T. Pang, H. J. T. Suh, L. Yang, and R. Tedrake, “Global planning for contact-rich manipulation via local smoothing of quasi-dynamic contact models,” IEEE Transactions on Robotics, vol. 39, no. 6, pp. 4691–4711, 2023. [39] M. Kiatos, I. Sarantopoulos, L. Koutras, S. Malassiotis, and Z. Doulgeri, “Learning push-grasping in dense clutter,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 8783–8790, 2022. [40] M. R. Dogar and S. S. Srinivasa, “A planning framework for nonprehensile manipulation under clutter and uncertainty,” Autonomous Robots, vol. 33, no. 3, pp. 217–236, 2012. [41] Y. Wang and H. Kasaei, “Learning dual-arm push and grasp synergy in dense clutter,” IEEE Robotics and Automation Letters, vol. 10, no. 5, pp. 5154–5161, 2025. [42] A. Escande, N. Mansard, and P.-B. Wieber, “Hierarchical quadratic programming: Fast online humanoid-robot motion generation,” The International Journal of Robotics Research, vol. 33, no. 7, pp. 1006– 1028, 2014. [43] S. Stavridis, P. Falco, and Z. Doulgeri, “Pick-and-place in dynamic environments with a mobile dual-arm robot equipped with distributed distance sensors,” in 2020 IEEE-RAS 20th International Conference on Humanoid Robots (Humanoids), 2021, pp. 76–82. [44] L. Koutras and Z. Doulgeri, “Exponential stability of trajectory tracking control in the orientation space utilizing unit quaternions,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021, pp. 8151–8158. [45] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Doll´ ar, and R. Girshick, “Segment anything,” 2023. [Online]. Available: https://arxiv.org/abs/2304.02643