Multi-Modal Fingertip for Contact Detection and Orientation Estimation

Multi-Modal Fingertip for Contact Detection and Orientation Estimation Olga Pennacchio Department of Engineering Universit` a degli Studi della Campania ”Luigi Vanvitelli” Aversa, Italy [email protected] Salvatore Pirozzi Department of Engineering Universit` a degli Studi della Campania ”Luigi Vanvitelli” Aversa, Italy salv[email protected] Abstract—Tactile and orientation information are very important to improve robotic manipulation capabilities in several contexts such as healthcare, industrial, logistic and domestic. This abstract aims at presenting a multi-modal sensing system that optimize accuracy through the integration of tactile sensors, an Inertial measurement unit and a connector for a Time-of-Flight pre-touch sensor. While tactile data enables closed-loop control during grasping operations, the measurements from the inertial unit are elaborated by state-of-the-art sensor fusion algorithms to obtain the sensor orientation in the 3D world. Index Terms—Optoelectronic Sensors; Tactile Sensing; Sensor fusion; Kalman filter; Madgwick filter, Mahony filter I. INTRODUCTION Currently, an ever-increasing number of robotic tasks is carried out in unstructured and cluttered environments, where the need for sensory systems is essential. In particular, touch, distance and orientation knowledge are of paramount importance in robotic manipulation and object grasping. Leveraging on the enhanced measurement accuracy and reduced noise that the combination of additional sensors provides, the sensor presented in this paper integrates a tactile sensing module, an Inertial Measurement Unit (IMU), and a Time-of-Flight (ToF) proximity sensor. The tactile solution represents an improved version of previous tactile sensor, whose characteristics can be found in [1], while details about ToF element and its capabilities can be found in [2]. In addition, the presence of an IMU allows to obtain an estimation of the orientation of the sensor by means of sensor fusion algorithms, i.e. Kalman, Madgwick, and Mahony. The first filter is based on a probabilistic model, while the other two on a quaternion-based representation, as described in [3]. This abstract shows the developed compact solution, specifically designed for mounting on the fingertip of robotic multi-fingered grippers, and its evaluation in robotic tasks. II. SENSOR DESIGN This section reports the sensor design, from both electronic and mechanical point of view. This work was supported by INTELLIMAN (G.A.No. 101070136), Horizon Europe, Research and Innovation Programme. 12.65 mm 25.90 mm Op�cal components MCU Programming Interface Connector Op-Amp Buffers Current channel DC/DC Converter IMU Microcontroller ToF Connector Fig. 1. Illustration of the PCB: top view (left) and bottom view (right). A. Electronic Section The sensor prototype has been arranged with a microcontroller-based Printed Circuit Board (PCB) shown in Fig. 1, housing an IMU and a tactile unit constitued by a matrix of ”taxels”, in which the base elements are the optoelectronic photo-reflectors (that are the couples of Light Emitting Diode and Photo-transistor), covered by a deformable layer. The tactile map is composed of spatially distributed measuring points, organized in a 4×3matrix, that work in reflection mode and return tactile data useful in several tasks, such as reconstructing the shape of object while in contact or estimating the applied forces. Furthermore, the presence of a 6−axis IMU, characterized by a 3−axis gyroscope and accelerometer and a temperature sensor, helps in continuous monitoring of its data, for example, to evaluate the pose, to avoid slippage events or to estimate the object temperature. Additionally, a specific connector on the sensor board allows the connection of the same microcontroller with a Time-of-Flight sensing element that can prevent collisions with obstacles, interact with the environment/human and reconstruct the 3D scene. B. Mechanical Section Regarding the mechanical part, the core element of the assembled sensor reported in Fig. 2 is the deformable pad. The latter has a flat surface as top side, and a bottom side characterized by parallelepiped shaped gaps with white ceilings, to boost reflection, and with black walls, to optically separate the taxels. The pad is manually manufactured in 2025 I-RIM Conference October 17-19, Rome, Italy ISBN: 9788894580570 10.5281/zenodo.17629646 63 Assembled sensor Case Microcontroller based PCB Rigid grid Deformable pad Exploded view Fig. 2. Model of the assembled sensor (left) and its parts (right). silicone (hardness equal to 20 Shore A), a material that shows better performance, with respect to other deformable materials, in terms of linear elastic property and low hysteresis. Between the deformable layer and the electronic board, there is a rigid grid that guarantees a monotonic working range for the photoreflectors. The case design aims to both enhance the resilience of the assembled sensor and to allows the mounting on the gripper fingers. In addition, the internal lateral surfaces of the case present some parallelepiped shaped protrusions that fasten the silicone during drying. The case as well as the rigid grid have been manufactured in Nylon by 3D printing technology. III. EXPERIMENTAL TEST To validate the sensor design presented in this work, a physical version has been manufactured and tested. The sensorized fingertip has been assembled on one finger of a commercial Robotiq 3-Finger Adaptive Gripper, that is mounted on an UR5e robot manipulator, to prove its proper integration as well as its use in grasping scenario and orientation estimation. To evaluate the tactile capability of the sensor, several grasp experiments have been performed with various objects of different shapes, texture and sizes, and the voltages coming from the twelve taxels are acquired during the tests. As an example, Fig. 3 displays those related to the pick of an eggplant, in which the contact between the fingertip and the grasped object is highlighted by the increasing taxels voltages after t= 50 s. Besides, the linear acceleration and the angular velocities, coming from the IMU, are combined to reconstruct 0 50 100 150 T ime [s] -0.02 0 0.02 0.04 0.06 0.08 0.1 V oltages [V] 1 2 3 4 5 6 7 8 9 10 11 12 Fig. 3. Tactile data during object grasping test. the orientation of the fingertip by means of sensor fusion algorithms, i.e. the Kalman filter, the Madgwick filter and the Mahony filter, having as output the Euler angles. To this aim, the robot is rotated around one sensor axis at a time and the data are acquired in each experiment. For instance, Fig. 4 shows an experiment in which the manipulator reorients the sensor, first rotating 90◦around the x-axis of the sensor (after t= 10 s) and then rotating −90◦around the same sensor axis (after t= 30 s). The sensor is positioned in such a way to align its x-axis with the q6sixth joint of the robot, so the value of the sixth joint can demonstrate the effective rotation. 3 4 q6[rad] -0.5 0 0.5 1 1.5 RP Y [rad] x y z -0.5 0 0.5 1 1.5 RP Y [rad] x y z 0 5 10 15 20 25 30 35 40 45 T ime [s] -0.5 0 0.5 1 1.5 RP Y [rad] x y z Fig. 4. Graph of the sixth joint value of the robot (first) and Euler angles from the Kalman (second), Madgwick (third) and Mahony (fourth) filter. IV. CONCLUSIONS AND FUTURE WORKS This work presents the design and test of a multi-modal sensor solution for fingertips of anthropomorphic grippers. The device has been used in several grasping scenarios to enable contact detection and orientation estimation. Future research will investigate model-based slippage control techniques using the multi-modal sensor solution presented in this paper. In particular, a possible way will be to use the tactile sensor for slippage control both in the presence of linear and rotational slippage. In addition, the IMU will be used to estimate the pose of the contact plane between the fingers and the objects, introducing also friction models between the manipulated objects. REFERENCES [1] A. Cirillo, M. Costanzo, G. Laudante, and S. Pirozzi, “Tactile sensors for parallel grippers: Design and characterization,” Sensors, vol. 21, no. 5, p. 1915, 2021. [2] A. Cirillo, G. Laudante, and S. Pirozzi, “Proximity sensor for thin wire recognition and manipulation,” Machines, vol. 9 (9), p. 188, 2021. [3] K. C¸ oc¸oli and L. Badia, “A comparative analysis of sensor fusion algorithms for miniature imu measurements,” International Seminar on Intelligent Technology and Its Applications, pp. 239–244, 2023. 64