Experimental Setup and Performance Assessment of a Wearable Embedded System for Human Hand Grasp Recognition via Differential Capacitive Sensing and Deep Learning

Experimental Setup and Performance Assessment of a Wearable Embedded System for Human Hand Grasp Recognition via Differential Capacitive Sensing and Deep Learning Francesca Federica Donadio ∗, Donatella Dragone ∗, Paolo Zaffino ∗, Carlo Cosentino ∗, Maria Francesca Spadea†, Francesco Amato‡, Gianluca Percoco§, and Alessio Merola∗, ∗Biomechatronics Laboratory, Universit` a Magna Græcia di Catanzaro, Campus “S. Venuta”, 88100 Catanzaro, Italy †Institute of Biomedical Engineering, Karlsruhe Institute of Technology, Karlsruhe, 76131, Baden-W¨ urttemberg, Germany ‡Dipartimento di Ingegneria Elettrica e delle Tecnologie dell’Informazione, Universit` a Federico II, 80125 Napoli, Italy §Department of Mechanics, Mathematics and Management, Polytechnic University of Bari, 70125 Bari, Italy Abstract—Over the years, research efforts in the field of hand gesture recognition have focused on developing novel algorithms and technologies, primarily based on inertial, optical, and electromyographic sensing. However, there is still a noticeable gap in providing truly wearable, energy-saving, embedded solutions for hand gesture recognition, which can benefit both from the adoption of novel capacitive sensing technologies and the implementation of machine learning algorithms for on-board and real-time recognition. This paper presents preliminary results on the design and implementation of a wearable embedded system that integrates, on a single glove, differential capacitive, dual-axis soft sensors, and an edge computing device to classify human hand grasps based on a recent taxonomy. Deep learning algorithms are implemented on a microcontroller-based platform to efficiently recognize four target hand grasping gestures. A dense 2-layer neural network is trained in the Edge Impulse machine learning environment and deployed on a low-cost board Arduino Nano 33 BLE Sense, achieving an accuracy of 87.5% in on-board recognition. Index Terms—wearable computing, differential capacitive sensing, human-robot interaction, hand rehabilitation, human grasp recognition I. INTRODUCTION Robot-assisted rehabilitation can successfully restore neuromotor functions and large-scale hand movements in patients with paralysis. However, recovering fine dexterity, such as individual finger motions and precise coordination required for effective manipulation tasks, remains difficult and typically requires prolonged and specialised training. Leveraging recent advances in soft robotics, robot-assisted hand rehabilitation systems, virtual reality-based therapies, soft exoskeletons with integrated pneumatic devices, and gloves with built-in sensors have been introduced to facilitate task-oriented training [1]. Embedded-sensing gloves facilitate the implementation of the digital interface on embedded systems, since they eliminate the need for complex front-end electronics and precise placement of electrodes, thus enabling fine-grained classification of types of hand grasp in real-time and on-board of the embedded computing device. The first quantitative taxonomy of hand grasps based on electromiography Corresponding author: C. Cosentino (email: [email protected]). (EMG) and kinematic data measurements has been proposed in [2], utilising a set of surface EMG electrodes with built-in accelerometers and a glove incorporating piezoelectric force and bending sensors. Among wearable sensing technologies useful for human grasp classification, the most promising are those based on capacitive principles and embedded in a soft dielectric elastomer support, offering, e.g., unlike piezoelectric-based solutions, superior conformability to the hand anatomy, enhanced sensitivity, and reduced power consumption. The efficiency of on-board inference performance for grasp recognition, which can be achieved through the implementation of machine learning models on off-the-shelf embedded computing platforms, is a trade-off among diverse design constraints, including inference accuracy, computation delay, and energy-saving features. Recently, capacitive sensing has been experimentally evaluated as a suitable modality for providing the minimal dataset required to train models and implement hand-gesture recognition on embedded systems, ensuring the expected inference performance (see, e.g., [3]). The available literature primarily addresses the recognition of gestures involving large-scale movements of the hand and wrist. In contrast, the recognition of fine-grasping tasks based on movements of the hand distal joints remains unexplored. To the best of the authors’ knowledge, this is the first work on machine-learning-based recognition and implementation on embedded computing devices of a series of grasp types, taken from the standard taxonomy in [2]. The deep learning model is developed through the Edge Impulse platform [4] and deployed on the Arduino Nano 33 BLE sense embedded platform, taking also the significant advantages of the differential capacitive 2-axis bending sensors Bend Labs (Nitto Bend Technologies, Inc.). II. EXPERIMENTAL SETUP The glove prototype, shown in Figure 1, integrates a two-axis Bend Labs sensor - characterized by a hyperelastic polymeric encasement that provides superior wearability and conformability to the finger motion - alongside the Arduino Nano board and two piezo-resistive bend sensors. Installed through knitted housings, this minimal set of sensors enabled the accurate classification of four hand grasp types, as defined by the standard taxonomy in [2]: parallel extension (PE), power disk (PW), large diameter (LD), and ring (R) (see Figure 2). The two-axis Bend Labs sensor offers the advantage of simultaneously capturing 2025 I-RIM Conference October 17-19, Rome, Italy ISBN: 9788894580570 10.5281/zenodo.17629702 83 d c a b Fig. 1. Glove prototype. (a) 2-axis soft capacitive sensor. (b)-(c) Piezoresistive bend sensors. (d) Arduino Nano board Fig. 2. Target hand grasps. A) parallel extension, B) power disk, C) large diameter and D) ring abduction-adduction and flexion-extension of the thumb, while piezoresistive bending sensors (Adafruit Inc.) measure flexion-extension of the index and middle fingers; the orientation angles of the hand (pitch and roll) are acquired through the Arduino-integrated IMU on the hand dorsum. The differential capacitive sensor outperforms other piezoresistive-based flex sensors, offering a compact solution for capturing motion along the thumb rotation axes, which are otherwise difficult to access. Indeed, a single Bend Labs sensor enables measurements on two independent axes with high sensitivity, linearity, and stable signal output, all while consuming very low power. Moreover, its differential configuration ensures robust performance even in the presence of electromagnetic interference and environmental variability. The Bend Labs sensor is connected to the Arduino Nano 33 BLE Sense Lite through I2Cchannel with a 3.3V input; the flex sensors’ signals are acquired on the analogue ports of the embedded platform. The availability on the Arduino Nano board of a floating point unit makes the embedded platform suitable for on-board elaboration of deep neural network models. III. RESULTS AND DISCUSSION The deep learning and embedded-computing implementation for achieving fine-grained recognition of grasps is obtained as a result of machine-learning models trained within the Edge Impulse [4] and tinyML [5] environments, providing a subset of machine learning models optimised to run on very low-power and microcontrollerbased embedded systems. The final stage of the modelling process involves porting the trained model onto the embedded microcontroller to achieve on-device inference directly. In this case, the on-board inference time is around 3 ms. The signals from the sensors on the glove are acquired as multi-channel time series (flex1, flex2, angle1, angle2, pitch, roll), which correspond to the data from the two piezo-resistive bending sensors, the 2-axis Bend Labs, and the integrated IMU, respectively. Data preprocessing is conducted using Edge Impulse’s Time Series Data block, employing 3000 ms windows with a step size of 1000 ms for the activation of a new window, Fig. 3. Confusion matrix with a sampling frequency of 10 Hz, to capture the dynamics of hand gestures. Feature extraction is performed using the Spectral Features block, which calculates parameters in the time and frequency domains. The generated features are used as input for a dense feedforward neural classifier, trained in Edge Impulse to learn distinctive gesture patterns and enable real-time classification on the Arduino Nano board. A repetition of 50 epochs has been judged adequate to obtain a well-trained model. The accuracy of the resulting model is satisfactory, with an accuracy of 87.5%. The classification confusion matrix provides validation of the proposed approach’s performance and is shown in Figure 3. IV. CONCLUSIONS AND FUTURE WORKS This work presents the main design steps and preliminary tests for validating a wearable embedded system that exploits capacitive sensing and machine learning for hand grasp recognition. The promising outcomes encourage further development to demonstrate the effectiveness of the proposed solution in integrating on-board inference on embedded devices for robot-assisted, virtual-realitybased hand rehabilitation, so that patients can perform therapy at home. ACKNOWLEDGMENT This work was supported by the Italian Ministry of University and Research through the PRIN 2022 PNRR programme - Project Title “3D Printed-Assembly free, controllable-compliance exosuits with distributed sensing for assistive and medical robotics”, code P20222WCJR - CUP: F53D23009530001 REFERENCES [1] H. Li and L. Cheng, “A systematic review on hand exoskeletons from the mechatronics aspect,” IEEE/ASME Transactions on Mechatronics, pp. 1–19, 2024. [2] F. Stival, S. Michieletto, M. Cognolato, E. Pagello, H. M¨ uller, and M. Atzori, “A quantitative taxonomy of human hand grasps,” Journal of NeuroEngineering and Rehabilitation, vol. 16, no. 1, Feb. 2019. [3] S. Bian and P. Lukowicz, “Capacitive sensing based on-board hand gesture recognition with tinyml,” in Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers, ser. UbiComp/ISWC ’21 Adjunct. New York, NY, USA: Association for Computing Machinery, 2021, p. 4–5. [4] (2025) Edge impulse. available online:. [Online]. Available: https://www.edgeimpulse.com/ [5] P. Warden and D. Situnayake, Machine Learning with Tensorflow Lite on Arduino and Ultra-Low-Power Microcontrollers. Sevastopol, CA, USA: O/’Reilly Media, 2019. 84