Real-time Haptic representation based on Tactile information

Real-time Haptic representation based on Tactile information 1st Christian Gianoglio DITEN, Unversity of Genoa Genoa, Italy [email protected] 2nd Chiara Micheli DITEN, Unversity of Genoa Genoa, Italy [email protected] 3rd Lucia Seminara DITEN, Unversity of Genoa Genoa, Italy [email protected] 4th Maurizio Valle DITEN, Unversity of Genoa Genoa, Italy [email protected] Abstract— Tactile sensing provides information about touch-based mechanical interactions with the environment. In this work, we present two complementary approaches for processing tactile data acquired through an artificial skin based on piezoelectric polymers. Keywords—Tactile sensing, data-driven, model-based, realtime data processing I. INTRODUCTION Despite their sophisticated mechanics, prosthetic hands still lack the dexterity of the human hands mainly due to the absence of haptic feedback. Likewise, in robotics, equipping artificial hands with a sense of touch is essential for enabling their dexterous interaction with the environment. Different types of tactile sensors generate heterogeneous data streams, possibly including spiking events. Each sensing system requires tailored methods to extract meaningful features such as contact location, force or pressure distributions, or properties of the contact surface, e.g. texture [1]. Processing these temporally varying, spatially distributed signals presents a major challenge. Both low-level information, such as slip detection and local force maps, and higher-level, cognitively relevant percepts, such as surface texture or object compliance, can be useful for sensorimotor control of artificial hands. Low-level signals may support rapid reflexive grip adjustments, while higher-level percepts may guide intentional manipulation and task planning. Integrating these information streams within a hierarchical control scheme can also be employed to restore dexterous, human-like sensorimotor function in prosthetics, not only robotics [2]. Building on these challenges, tactile signals from electronic skins can be both processed through physics-based models and AI-driven strategies, which must engage in a bidirectional dialogue to mutually inform perception and control. This work outlines two promising methodologies— model-based and data-driven—for extracting haptic representations from tactile signals acquired by piezoelectric sensor arrays embedded in a soft layer. Furthermore, it reflects on future perspectives and challenges, including the need to create synergies between the two approaches to open novel routes for closing the sensorimotor control loop in prosthetics and robotics. II. DISTRIBUTED ELECTRONIC SKIN Our electronic skin (ES) is made of piezoelectric PVDFTrFE (polyvinylidene fluoride-trifluoroethylene) transducers, covered with a widely-used soft protective layer (Dragon skin). PVDF transducers are capable of measuring dynamic contacts and transient events, effectively covering the entire frequency range of all human mechanoreceptors, except for measuring static stimuli. The embedded electronics (EE) [3], equipped with the ARM Cortex-M7 microcontroller, acquires up to 64 channels at a sampling frequency of 2 kHz. Processing algorithms can also be deployed directly on the EE, enabling real-time data analysis and the extraction of haptic information, which can be used either to drive autonomous control strategies or to provide feedback to the user. III. HAPTIC REPRESENTATION Haptic representations encode diverse object and contact attributes — such as geometry (shape, edges, orientation), surface properties (e.g., texture), bulk mechanical properties (e.g., stiffness/compliance), and dynamic contact cues (e.g., slip) — by integrating tactile and proprioceptive signals. In model-based processing, contact information is estimated from tactile sensor outputs through physics-based or analytical models. Alternatively, machine learning can be employed to directly infer high level haptic representations from raw sensor data, bypassing explicit model inversion. A. Model-based Approaches In [4], a model was presented to predict the sensor response to a given force or reconstruct the contact force from sensor output, in static conditions. The model evaluates how the ES protective layer transmits distributed normal (frictionless case) and tangential forces (frictional case) to an extended PVDF transducer. The model combines FEM simulations with an analytical formulation and accounts for forces applied both centrally and off-center with respect to the transducer. While such model-based framework requires stringent assumptions and simplifications, it offers a possible route to reconstruct magnitude and direction of the contact force and its position (from the transducer outputs). Future work will be needed to assess whether the model can reconstruct the dynamic evolution of contact forces or contact area, and to explore its optimization for real-time embedded implementation. Moreover, our previous model [5] is still promising to reconstruct the whole force distribution from the outputs of the sensor array. The feasibility of a real-time embedded implementation of this algorithm on FPGA platforms has already been assessed for the static case; however, further work is required to extend this approach to 2025 I-RIM Conference October 17-19, Rome, Italy ISBN: 9788894580570 10.5281/zenodo.17629842 207 the dynamic case. Reconstructing the temporal evolution of contact forces is also valuable for real-time detection of the onset of slip, and the presented algorithms outline promising avenues for further investigation. B. AI-based Approaches In our research group, AI algorithms have been developed to extract object properties from tactile data. These algorithms were designed for deployment on embedded systems, and some have already been tested on such platforms. In [6][7], the softness of 3D-printed objects was classified. Specifically, in [6], deep neural networks implemented on the EE were used to classify five levels of softness from data acquired by an ES composed of eight sensors mounted on the Baxter robot’s end effector. In [7], a Cartesian robot equipped with the ES, consisting of eight sensors encapsulated within a biomimetic finger, performed indentation tests on 3D-printed objects. Tactile data have been collected through the EE and sent to the PC via USB connection. Features extracted from the acquired signals were ranked using the PCA biplot technique, reduced, and subsequently employed to train and evaluate a support vector machine (SVM), a single-layer neural network (SLNN), and a k-nearest neighbor classifier. In [8][9], texture recognition was performed on both 3D-printed and naturalistic surfaces. In particular, [8] employed neuromorphic computing to encode and classify signals acquired from the ES, consisting of eight sensors encapsulated within a biomimetic finger mounted on a Cartesian robot. Sliding actions were performed over eight 3D-printed textures, and tactile data were collected via the EE and transmitted to a PC. In [9], the ES was mounted on a glove worn by seven participants performing sliding actions on six naturalistic textures. Tactile data have been collected by the EE and sent to the PC. Features were extracted from the signals in the time and frequency domains to train a support vector machine (SVM) and a single-layer neural network (SLNN). Additionally, raw data were used to train a 1D convolutional neural network (CNN). Recently, our ESs have been developed for both robotic and prosthetic applications. Two ES patches, each consisting of 16 sensors, were designed for the end effector of a Tiago robot. In an initial study, the robot was programmed to grasp various objects while signals were recorded using the EE. AI techniques were applied for real-time object recognition: the signals were either windowed and fed into deep neural networks or processed to extract features for training a support vector machine (SVM). Additionally, Hannes and AR10 hands were fully sensorized using removable solutions consisting of five fingertips. The objective of this research is to develop AI algorithms deployable on the EE to detect slippage for autonomous hand control and to extract haptic properties of the manipulated objects. Moreover, building on the results of [10][11], optimization techniques for designing AI algorithms that meet the constraints of embedded devices are under development. In particular, neural architecture search (NAS) is being employed to design 1D-CNNs while accounting for the hardware limitations of the EE. IV. FUTURE PERSPECTIVES AND CHALLENGES The progress achieved in extracting meaningful information from tactile data through both model-based and AI-driven approaches represents a first step toward restoring the sense of touch in prosthetic systems and closing the sensorimotor control loop in robotics. Nevertheless, some challenges remain open. Physics-based models provide interpretable insights into the mechanics of tactile interaction, yet they inevitably rely on assumptions and simplifications that limit their applicability to real-world scenarios. Conversely, AI-based approaches excel at capturing complex, nonlinear patterns directly from sensor data, but they require large-scale, high-quality datasets and often struggle to generalize across different contexts, sensors, or environmental conditions. Moreover, real-time processing remains a major challenge for both approaches: our physical model can be computationally demanding in its dynamic evolution, while AI-based methods deployed on the EE can extract only one haptic representation at a time, creating a bottleneck when multiple types of information are required. Bridging this gap will require hybrid strategies that combine the interpretability of physical models with the adaptability of AI, ensuring both effectiveness and robustness in real environments. Ultimately, the main bottleneck still remains the ability to provide intuitive feedback to the user in human-in-the-loop control systems, calling for systems that can integrate multimodal information and can learn and adapt through experience. ACKNOWLEDGMENTS This work was supported by the IntelliMan project: EU H2022, under Grant 101070136. REFERENCES [1] Luo, S., Lepora, N. F., Yuan, W., Althoefer, K., Cheng, G., & Dahiya, R. (2025). Tactile Robotics: An Outlook. IEEE Transactions on Robotics. [2] Seminara, et al. (2023). A hierarchical sensorimotor control framework for human-in-the-loop robotic hands. Science Robotics, 8(78), eadd5434. [3] Abbass, Y., Saleh, M., Dosen, S., & Valle, M. (2021). Embedded electrotactile feedback system for hand prostheses using matrix electrode and electronic skin. IEEE Transactions on Biomedical Circuits and Systems, 15(5), 912-925. [4] Micheli, C., Berselli, G., & Seminara, L. (2024). Model-based approach for indentation on soft electronic skin. Mechatronics, 102, 103225. [5] Seminara, L., Capurro, M., & Valle, M. (2015). Tactile data processing method for the reconstruction of contact force distributions. Mechatronics, 27, 28-37. [6] Amin, Y., Gianoglio, C., & Valle, M. (2023). Embedded real-time objects’ hardness classification for robotic grippers. Future Generation Computer Systems, 148, 211-224. [7] Bassal, H., Abbass, Y., Gianoglio, C., & Valle, M. (2024). Hardness discrimination using piezoelectric-based biomimetic tactile sensor and machine learning. IEEE Sensors Letters. [8] Ali, H. A. H., Abbass, Y., Gianoglio, C., Ibrahim, A., Oh, C., & Valle, M. (2024). Neuromorphic tactile sensing system for textural features classification. IEEE Sensors Journal, 24(10), 17193-17207. [9] Abbass, Y., Gianoglio, C., Ali, H. A. H., Saleh, M., & Valle, M. (2024). Texture perception using tactile sensing glove based on PVDF sensors and machine learning. IEEE Sensors Letters, 8(7), 1-4. [10] Gianoglio, C., Ragusa, E., Gastaldo, P., & Valle, M. (2021). A novel learning strategy for the trade-off between accuracy and computational cost: a touch modalities classification case study. IEEE Sensors Journal, 22(1), 659-670. [11] Gianoglio, C., Ragusa, E., Gastaldo, P., & Valle, M. (2023). Trade-off between accuracy and computational cost with neural architecture search: A novel strategy for tactile sensing design. IEEE Sensors Letters, 7(5), 1-4. 208