Vision-based Robotic Arm Control and Tactile Sensing for Dice Manipulation

Vision-based Robotic Arm Control and Tactile Sensing for Dice Manipulation Tommaso Faraci1* Anna Labardi2* Hamidreza Raei3†* Mianzhi Song3†* Zhengyin Du 4†* Abstract—Robust and precise object manipulation remains a central challenge in robotics, requiring the tight integration of perception and control. In this work, a robotic system capable of recognizing and reorienting a dice to a desired face by combining computer vision and tactile sensing to aid robotic manipulation is presented. Index Terms—robotic demonstration, DRIMS2, manipulation, computer vision, tactile sensing I. INTRODUCTION This work presents the integration of mechanical design, tactile sensing, computer vision, and control strategies [1] within a robotic demonstration developed during the 3rd Doctoral Summer School on Robotics and Intelligent Machines (DRIMS2). The goal of the challenge was to design and implement a solution where a robotic arm, equipped with a custom gripper embedding tactile sensors, could localize and manipulate a dice according to a specified objective [2]. Moreover, the system had to recognize the dice configuration and autonomously reorient it until a certain pre-specified face was correctly positioned upwards (Figure 1). II. MATERIALS AND METHODS The experimental setup consisted of a commercial robotic manipulator (UR10, Universal Robots) equipped with a custom gripper, an overhead OAK-D Pro camera (Luxonis, USA) framing a simple 6-faced dice on a table, and two tactile pads integrating Fiber Bragg Grating (FBG) sensors [3] fitted around the gripper clamps. The camera provided real-time information on the dice upper face and pose [4], enabling the robot to plan the necessary grasps and rotations to complete the challenge, while the FBG sensors allowed continuous monitoring of the contact conditions to ensure stable manipulation. A. Gripper Design The gripper was designed in Creo Parametric (PTC, USA), with the main constraints being: compatibility of the sensor installation area with the sensor pads, adaptability of the connection base to the sliders of the Hand-E Adaptive Gripper (Robotiq), and sufficient clamp spacing to accommodate the dice. The component was fabricated in Polylactic Acid using a Fused Deposition Modeling Bambu Lab P1S 3D printer (Bambu Lab, China). 1Dept. of Information Eng. and Computer Science, Univ. of Trento, Trento, Italy - [email protected].2The Biorobotics Institute, Scuola Superiore Sant’Anna, Pisa, Italy - anna.labar[email protected].†Istituto Italiano di Tecnologia, Genova, Italy. 3Human-Robot Interfaces and Interaction - hamidr[email protected] & [email protected] 4Soft BioRobotics Perception - [email protected].*The authors contributed equally to this work. Fig. 1. Dice manipulation during the challenge. B. Tactile Sensing Two silicone pads embedding optical fibers with ten FBG sensors each were fitted around the gripper clamps. The fibers were connected to an optical interrogator (FBG-Scan 904, FBGS) acquiring and streaming data at 100 Hz. A LabVIEW (National Instruments, USA) Graphical User Interface (GUI) was implemented to receive the data, perform signal recalibration by offset removal, and compute both the norm and the median derivative of the ten sensor values per finger. Empirically defined thresholds were applied: a norm exceeding the threshold on both fingers indicated grasp detection, while a median derivative above threshold signaled either force transients or slippage. This setup enabled continuous monitoring of contact conditions, including grasp detection, slippage identification, and adaptive force regulation. C. Computer Vision This section outlines the perception pipeline used to estimate the dice pose relative to the robotic arm, detect the top face, and identify its number. The workflow overview is shown in Fig. 2. The RGB camera feed undergoes morphological transformations exploiting the color contrast between the dice and background. After masking, the dice contour is extracted with perspective correction. Circular dots on the top face are detected through opening, dilation, and erosion, followed by ellipse fitting to handle projection effects and remove artifacts [5]. Opening mitigates reflections and surface defects, while erosion enhances dot separation. Ellipse fitting identifies dots by thresholding eccentricity. The face center is computed as the mean of ellipse centroids cell: pcentroid =1 Ndots Ndots X i=0 cell,i (1) An oriented bounding box around the contour defines the dice pose in the image as (pcentroid, θ)after correcting perspec2025 I-RIM Conference October 17-19, Rome, Italy ISBN: 9788894580570 10.5281/zenodo.17629724 99 Fig. 2. Depiction of the sequence of transformations and operations undergone by the RGB image to detect the die pose in the image. tive distortion. Since the die rests on the green screen before grasping, the depth zCD is known, allowing its 3D position tCD to be computed using the pinhole camera model [6]. The world-frame pose TWD is then obtained via TWD =T−1 CWTCD, where TCD combines Rz(θ)and tCD. The pipeline delivers reliable poses to perform grasping. The method has been tested on prerecorded datasets as well as in real grasping experiments. D. Control Strategy The pose information provided by the vision pipeline was used to control a position-based manipulator. Since selfocclusion occurs during grasping, the strategy aimed to minimize pose-estimation calls in both grasp and rotation planning. The proposed motion-planning strategy, illustrated in Fig. 3, required at most three observations to achieve the transition from an initial face Fto a desired face D, as shown in (2). To minimize the number of the dice pose estimation (Observation) the algorithm would assume a rigid contact between end-effector and the dice, once the gripper was closed (the dice is grasped), and the dice position (and not orientation) was kinematically propagated by attaching it to the endeffector frame. Re-estimation of the dice pose through vision algorithm is deferred until next observation if required by the control strategy shown in Fig. 3. To mitigate unintended rolling, gripper open/close actions were executed close to the tabletop, thereby reducing perception latency and overall task duration. The estimation of maximum number of observation to reach the task objective is indicated in the following: P(O= 1) = 1 3,P(O= 2) = 1 3,P(O= 3) = 1 3.(2) For a uniformly random initial top face, the expected number of observations was two, as detailed below: E[O|F, D] = 1 + 1! [F /∈D, , 7−D]·3 2,(3) E[O] = 1 ·1 3+ 2 ·1 3+ 3 ·1 3= 2.(4) III. RESULTS AND DISCUSSIONS The proposed system successfully executed the dice manipulation task, reorienting the dice to the desired face in at most three moves with the proposed control strategy. However, face detection depends on predefined color segmentation for both dice and background, limiting generalization to other environments or dice types. During manipulation, arm occlusions may block visual feedback, so the robot must periodically reposition itself to re-establish the line of sight and re-identify the dice pose after each manipulation step. Fig. 3. Control strategy: ”F” stands for ”Face”, ”P” stands for ”Pose” and ”D” stands for ”Desired face”. IV. CONCLUSIONS This work presented a robotic system capable of autonomously reorienting a dice, highlighting the potential of integrating multimodal perception and planning for reliable manipulation tasks. A commercial robotic manipulator equipped with a custom 3D-printed gripper and two tactile pads embedding FBG sensors was used to manipulate a dice. A camera provided real-time information on the dice face and pose, while dedicated algorithms for vision, sensor processing, and control enabled autonomous planning of dice grasps and reorientations. Overall, the system successfully completed the manipulation task both in simulation and in real experiments, achieving the desired dice face in at most three moves under controlled visual and lighting conditions. V. ACKNOWLEDGEMENTS This study was carried out within the 3rd Doctoral Summer School on Robotics and Intelligent Machines (DRIMS2). The authors thank the organizing committee for providing the platforms and the support that made this work possible. REFERENCES [1] T. Watanabe, K. Yamazaki, and Y. Yokokohji, ”Survey of robotic manipulation studies intending practical applications in real environments - object recognition, soft robot hand, and challenge program and benchmarking”, Advanced Robotics, 2017. [2] B. Siciliano, L. Sciavicco, L. Villani, and G. Oriolo, ”Robotics: Modelling, Planning and Control”, Springer, 2009. [3] T. Erdogan, ”Fiber grating spectra,” Journal of Lightwave Technology, 1997. [4] R. Szeliski, “Computer vision: Algorithms and applications,” Springer, 2011. [5] F. Y. Shih, Image Processing and Mathematical Morphology. Boca Raton, FL: CRC Press, 2009. [6] R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed. Cambridge: Cambridge University Press, 2004. 100