Resource-Efficient GRU for Real-Time Gesture Recognition

Resource-Efficient GRU for Real-Time Gesture Recognition Michail Kaseris* International Hellenic University Katerini, Greece [email protected] *Corresponding author Sotiris Malassiotis Information Technologies Institute CERTH Thessaloniki, Greece [email protected] Ioannis Kostavelis International Hellenic University Katerini, Greece gkosta[email protected] Abstract—Time-of-flight sensors deliver distance measurements through light propagation timing, facilitating object detection, gesture recognition, and three-dimensional mapping. These sensors maintain functionality across lighting conditions while ensuring response rates required for real-time applications. Based on these properties, combined with their cost and energy consumption parameters, we propose their exploitation in gesture recognition systems for human-robot interaction (HRI). This work presents two contributions: First, we introduce a dataset for gesture recognition tasks. Second, we develop a model for sequence classification optimized for resource constraints. The experimental results demonstrate the feasibility of the approach through quantitative evaluation metrics. Finally, we present classification results on the dataset, analyze the model’s performance on gesture categories, and discuss future challenges and directions in this field of research. Index Terms—Time-of-Flight Sensors, Gesture Recognition, Human-Robot Interaction I. INTRODUCTION Hand gesture recognition has expanded beyond everyday applications into diverse fields including virtual reality, medical systems, education, and automotive interfaces. Among the three main recognition technologies—data glove-based, vision-based, and radar-based—data gloves offer precise tactile sensing but present limitations in practicality. Despite their advantage in not requiring gesture extraction from background scenes, data gloves face adoption barriers due to their bulk, high costs, and calibration requirements, making them less widely applicable than vision-based alternatives [1], [2]. Time-of-Flight (ToF) cameras produce depth images by encoding scene distances in individual pixels through phasedelay measurements of reflected infrared light. This technology enables direct 3D structure estimation while exhibiting robustness to radiometric, geometric, and illumination factors. Other benefits include compact design, CMOS-based costeffectiveness, high frame rates, and measurement accuracy. ToF cameras have proven valuable across diverse applications including robot navigation, 3D reconstruction, and humanmachine interaction, offering a complementary approach to This research was funded by the European Union’s Horizon Europe Project “Sestosenso” (http://sestosenso.eu/ (accessed on 27 November 2024), HORIZON-CL4-Digital-Emerging) under Grant 101070310. other depth-sensing methods such as structured-light systems [3]. In the context of human computer interaction, gesture recognition by means of 3D sensors (such as Kinect) has been already investigated [9], also in robotics applications [10]. The main downside of such systems is that they require significant computational resources e.g. for human skeleton extraction or hand pose estimation. They also require most of the user body to be visible by the camera. Thus they are suitable in use cases where a fixed camera is looking directly towards a user with limited movement and minimal occlusions. On the contrary our application target are use cases such as collaborative assembly in automotive industry where fixing the camera is not possible e.g. due to significant occlusion of the workspace and the workers movement is significant. In such use cases we are experimenting with two solutions one based on a network of body warn IMU sensors and the second, presented in this paper, based on a ring of ten low-cost/lowresolution ToF sensors in a multi-view setup integrated on the collaborative robot manipulator. By exploiting multiple sensors with different viewpoints we achieve robustness with respect to robot and user movement. In addition the proposed approach has very low latency and computational requirements while achieving high accuracy with a relatively small number of gestures despite the fact that the sensors have a very low resolution (8×8). User feedback may also be conveniently provided by means of colored LEDs or audio integrated on the ring. In summary this work contribution to the field of humanrobot interaction are: •Demonstration and evaluation of gesture recognition based on a novel multi-view low resolution ToF sensor setup. •Capturing a new dataset of several gesture sequences captured by the afformentioned ToF sensors array. •Building upon this dataset, we propose a resourceefficient classification model for sequence classification that addresses computational constraints. 66 2025 11th International Conference on Automation, Robotics, and Applications 979-8-3315-0923-1/25/$31.00 ©2025 IEEE 2025 11th International Conference on Automation, Robotics, and Applications (ICARA) | 979-8-3315-0923-1/25/$31.00 ©2025 IEEE | DOI: 10.1109/ICARA64554.2025.10977711 Authorized licensed use limited to: Universidad de Zaragoza. Downloaded on September 30,2025 at 11:10:41 UTC from IEEE Xplore. Restrictions apply. II. RELATED WORK The work proposed by [5] presents a gesture recognition system for UAV control using a data representation model that transforms 4D spatiotemporal data into 2D matrices and 1D arrays. Their system processes skeleton data from a Leap Motion Controller and employs three neural network architectures: a 2-layer fully connected network, a 5-layer fully connected network, and an 8-layer convolutional network. The approach was validated both in simulation and on physical drone platforms, demonstrating the feasibility of gesturebased drone control through their proposed data representation model. [6] addresses real-time performance limitations in UAV control through a hybrid system combining IMU and vision-based gesture recognition. Their approach uses a thumb-mounted IMU for continuous commands and visionbased detection for discrete inputs, overcoming challenges in dynamic gesture recognition that typically affect purely vision-based systems. The framework demonstrates improved performance over traditional control interfaces in simulation environments. [7] propose a PointNet-based deep neural network for hand gesture recognition using ToF sensor data. Their method includes a multistage hand segmentation process and demonstrates that 3D point cloud processing outperforms 2D approaches. The work includes a custom dataset creation and comparative analysis between 2D and 3D methodologies. [12] developed a real-time gesture recognition system utilizing time-of-flight camera data for Windows application control. Their approach combines morphological analysis of hand silhouettes with motion pattern estimation to recognize static and dynamic gestures, demonstrating an effective alternative to conventional interfaces. III. PROPOSED APPROACH A. Hardware Setup The experimental setup employs a Time-of-Flight (ToF) sensor ring array comprising 10 ToF cameras [11], strategically positioned at varying distances (20cm to 1m) from the subject, as shown in figure 1. This distance range was specifically chosen to reflect realistic human-robot collaboration scenarios, where operators typically work in close proximity to the robot during task execution. The cameras are arranged in a uniform circular distribution within a custom-designed chassis, with each sensor positioned at equal intervals of 36 degrees (360°/10) to ensure complete coverage of the surrounding space, as depicted in figure 2. This 360-degree receptive field is essential as the operator’s position relative to the robot is not fixed and can vary during operation. Each ToF sensor captures depth data at a resolution of 8x8 pixels, creating a compact yet informative representation of the gesture space. The synchronization of the multi-camera system is handled through a buffer-based approach: the system accumulates frames from all sensors, and once a complete set is received, a unified timestamp is assigned to the entire frame collection by the publisher before proceeding to the next capture cycle. This configuration enables comprehensive capture of gesture data from multiple viewpoints simultaneously. Fig. 1. Our experimental setup. Time-of-Flight Sensor Fig. 2. A top view of the sensor array. Each small circle represents a Timeof-Flight sensor. All sensors are distributed equally on the circumference of the circular base. The red triangles represent the field of view of each sensor. A gesture can take place in front of any time-of-flight sensor. B. Dataset Acquisition and Preprocessing The gesture recognition system employs a deep learning approach to process sequential gesture data, implementing and comparing both Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) neural network architectures. Each frame of input consists of a 640-dimensional feature vector, derived directly from the raw sensor array data (10 sensors × 8×8 pixels). The network architecture consists of a single-layer recurrent network with 128 hidden units, connected to a linear output layer that maps to gesture classes. The hidden unit size of 128 was empirically determined during initial experiments and demonstrated stable performance across different gesture patterns. 67 Authorized licensed use limited to: Universidad de Zaragoza. Downloaded on September 30,2025 at 11:10:41 UTC from IEEE Xplore. Restrictions apply. Data preprocessing follows a straightforward approach: raw sensor readings are rescaled from their original range [0, 4000] to [0, 1] through min-max normalization. To facilitate efficient batch processing, sequences are zero-padded to a uniform length, ensuring consistent input dimensions across all samples. C. Neural Network Architecture The network architecture consists of a Gated Recurrent Unit (GRU) [8] followed by a linear classification layer. The input to the network is a sequence of 640-dimensional vectors, where each vector represents the concatenated readings from all ToF sensors at a single time step. These sequences have variable lengths depending on the duration of the performed gesture. To handle variable-length sequences efficiently, the input is first processed using packed sequences, which allows the GRU to operate only on actual data points while ignoring padding. The GRU processes these sequences with hhidden units across llayers, maintaining a hidden state that captures temporal dependencies in the gesture data. The initial hidden state h0is initialized as a zero vector. For classification, the network utilizes only the final relevant output of each sequence (corresponding to the last actual time step, excluding padding). This output is then passed through a linear layer that maps the h-dimensional hidden representation to a probability distribution over the gesture classes. Formally, for an input sequence (X= (x1, ..., xT)) where (xt∈R640), the network computes: ht=GRU(xt, ht−1)(1) y=W hT+b(2) where htrepresents the hidden state at time t,Wand bare the weights and bias of the final linear layer, and yis the probability vector used for classification. IV. EXPERIMENTS A. Dataset A dataset of hand gestures was collected using a Time-ofFlight sensor array. Two participants performed three distinct gestures: ”come here,” ”stop,” and ”spin.” To introduce variability in the data collection, the gestures were executed from both standing and seated positions at varying distances from the sensor array. The dataset includes variations where participants alternated between their dominant and non-dominant hands, as well as instances of two-handed gesture execution. In total, 80 sequential hand gesture samples were recorded, providing a diverse collection of spatiotemporal data for gesture recognition analysis. B. Implementation Details The experimental implementation utilized PyTorch for model development and training. The GRU-based classifier was trained for 50 epochs with a batch size of 32 and a learning rate of 0.001 using the Adam optimizer. The model architecture consisted of two GRU layers with 128 hidden units, followed by a fully connected layer for classification. The input dimension was set to match the concatenated ToF sensor data (64 dimensions ×10 sensors), and the output dimension corresponded to the three gesture classes: ”come here,” ”spin,” and ”stop.” The dataset was systematically divided into training and testing sets, with three samples per gesture class reserved for testing to ensure a balanced evaluation. To handle variable-length sequences efficiently, we implemented a custom collate function that sorted sequences by length and applied padding where necessary. This approach optimized batch processing while preserving the temporal characteristics of the gesture data. C. Training Protocol The training process employed cross-entropy loss for classification and incorporated dynamic sequence packing to accommodate variable-length gestures. We monitored both training loss and validation accuracy throughout the training process, saving model checkpoints based on best validation performance. The batch-first data format was utilized to optimize memory usage and computational efficiency during training. D. Results The experimental results demonstrated the effectiveness of our approach in recognizing the three distinct gesture classes. The model achieved consistent performance across different gesture types, with variations in recognition accuracy correlated with gesture complexity. The confusion matrix visualization revealed patterns in classification errors, providing insights into potential areas for improvement in both data collection and model architecture, as illustrated in figure ??. Training metrics showed steady convergence, with the loss function decreasing monotonically and validation accuracy stabilizing after approximately 30 epochs. This behavior suggested that the chosen hyperparameters and model architecture were wellsuited to the gesture recognition task. The final model achieved robust performance on the test set, demonstrating its capability to generalize to unseen gesture instances while maintaining real-time processing capabilities necessary for human-robot interaction applications. E. Model Architecture Evaluation We performed a systematic evaluation of various neural network architectures to determine the optimal configuration for gesture recognition. The analysis examined the effects of recurrent unit type (GRU, LSTM), hidden state dimensionality (64, 128, 256), network depth (1-3 layers), and regularization strength (dropout rates: 0.0, 0.2, 0.4). Figure 3 presents the quantitative results of this investigation. 1) Performance Analysis: The empirical results indicate superior performance of GRU-based architectures over LSTM variants, with mean validation accuracy of 88-90% and 67% respectively. GRU configurations also exhibited reduced variance across experimental conditions, indicating enhanced stability. Analysis of hidden state dimensionality revealed a 68 Authorized licensed use limited to: Universidad de Zaragoza. Downloaded on September 30,2025 at 11:10:41 UTC from IEEE Xplore. Restrictions apply. Fig. 3. Hyperparameter performance analysis of the classifier. TABLE I PERFORMANCE ANALYSIS OF GRU CLASSIFIER Sequence CPU Performance GPU Performance Length Time (ms) Memory (MB) Time (ms) Memory (MB) 50 2.86 0.89 0.70 0.85 100 5.08 0.93 1.08 8.98 150 7.43 0.96 1.68 8.98 200 9.95 1.00 1.97 8.98 250 12.42 1.04 2.45 8.98 300 14.87 1.07 2.88 8.98 350 17.17 1.11 3.32 8.98 400 19.36 1.15 3.87 8.98 positive correlation with model performance, where representations of dimension 256 achieved optimal accuracy with minimal variance. The investigation of architectural depth produced an unexpected finding: single-layer configurations systematically outperformed deeper architectures in both accuracy and consistency. This observation suggests that the temporal dynamics of gesture recognition may be sufficiently captured by a single recurrent layer. Furthermore, the application of dropout regularization demonstrated an inverse relationship with model performance. The absence of dropout yielded superior results, indicating that the constraint of network capacity may be detrimental for this particular recognition task. As shown in Table I, the model demonstrates efficient scaling with sequence length, with GPU inference time increasing linearly from 0.70ms at 50 frames to 3.87ms at 400 frames, while maintaining consistent memory usage of approximately 9MB. CPU inference exhibits similar linear scaling but at higher absolute times, ranging from 2.86ms to 19.36ms. These results indicate that real-time gesture recognition is feasible on both platforms, with GPU acceleration providing a 4-5x speedup while maintaining minimal memory overhead. Through this systematic evaluation, we determined that optimal performance is achieved using a single-layer GRU architecture with hidden state dimension 256 and no dropout regularization. This configuration demonstrates that architectural simplicity, combined with sufficient representational capacity, provides robust performance for gesture recognition. The results suggest that the temporal structure of gesture data may be effectively modeled without requiring deep architectural hierarchies. V. CONCLUSION This work presented a gesture recognition system using Time-of-Flight sensors for human-robot interaction applications. Through experimental evaluation, we demonstrated the effectiveness of a circular ToF sensor array configuration coupled with a single-layer GRU architecture, achieving 8890% accuracy across three gesture classes. Our performance analysis showed that the system can process gesture sequences efficiently on both CPU and GPU platforms, with inference times scaling linearly with sequence length. Key findings include the superiority of GRU over LSTM architectures and the unexpected effectiveness of single-layer configurations. Future work should focus on expanding the gesture dataset and validating the system in real-world human-robot interaction scenarios. This work contributes to the field by demonstrating that resource-efficient architectures can achieve robust performance when combined with strategically positioned Time-ofFlight sensors. REFERENCES [1] Suarez, J., Murphy, R. R. (2012, September). “Hand gesture recognition with depth images: A review”. In 2012 IEEE RO-MAN: the 21st IEEE international symposium on robot and human interactive communication (pp. 411-417). IEEE. [2] Khan, R. Z., Ibraheem, N. A. (2012). “Hand gesture recognition: a literature review”. International journal of artificial Intelligence & Applications, 3(4), 161. [3] Hansard M., Lee S., Choi O., Horaud R. “Time of Flight Cameras: Principles, Methods, and Applications.” Springer, pp.95, 2012, SpringerBriefs in Computer Science, ISBN 978-1-4471-46582. 10.1007/9781-4471-4658-2 [4] Li, L. “Time-of-flight camera—an introduction.” Technical white paper SLOA190B (2014). [5] Hu, B., Wang, J. (2020). “Deep learning based hand gesture recognition and UAV flight controls.” International Journal of Automation and Computing, 17(1), 17-29. [6] Yoo, M., Na, Y., Song, H., Kim, G., Yun, J., Kim, S., Jo, K. (2022). Motion estimation and hand gesture recognition-based human–UAV interaction approach in real time. Sensors, 22(7), 2513. [7] Mirsu, R., Simion, G., Caleanu, C. D., Pop-Calimanu, I. M. (2020). “A pointnet-based solution for 3D hand gesture recognition.” Sensors, 20(11), 3226. [8] Cho, K. (2014). “Learning phrase representations using RNN encoder-decoder for statistical machine translation”. arXiv preprint arXiv:1406.1078. [9] S. Malassiotis, N. Aifanti and M. G. Strintzis, ”A gesture recognition system using 3D data,” Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission, Padua, Italy, 2002, pp. 190-193 [10] Alexeev, Alexander et al. “An Overview of Kinect Based Gesture Recognition Methods.” Proceedings of International Conference on Artificial Life and Robotics (2024) [11] VL53L8CX - Low-power high-performance 8x8 multizone Time-of-Flight sensor (ToF) https://www.st.com/en/ imaging-and-photonics-solutions/vl53l8cx.html [12] Molina, J., Escudero-Vi˜ nolo, M., Signoriello, A., Pard` as, M., Ferr´ an, C., Besc´ os, J., ... & Mart´ ınez, J. M. (2013). Real-time user independent hand gesture recognition from time-of-flight camera video using static and dynamic models. Machine vision and applications, 24, 187-204. 69 Authorized licensed use limited to: Universidad de Zaragoza. Downloaded on September 30,2025 at 11:10:41 UTC from IEEE Xplore. Restrictions apply.