Towards Exploiting EEG in Robot Teleoperation: A Systematic Dataset Protocol for Error-Related Potentials Across Cognitive Load Levels

Towards Exploiting EEG in Robot Teleoperation: A Systematic Dataset Protocol for Error-Related Potentials Across Cognitive Load Levels Nicole Alati, Elisa Magosso, Gianluca Palli, and Roberto Meattini Abstract—Error-Related Potentials (ErRPs) in brain activity can provide insight into user intent prediction errors during robot teleoperation. This paper proposes a protocol for systematically acquiring EEG recordings from participants performing simplified virtual teleoperation and telemanipulation tasks with increasing cognitive load, including artificially induced error trials. With synchronized task triggers and commands, the protocol enables analysis of how cognitive load affects the occurrence, amplitude, and timing of ErRPs, supporting research in adaptive human-robot interaction. A preliminary dataset from five subjects following the proposed protocol is made available to the community. Index Terms—Error-Related Potentials (ErRPs), Cognitive Load, Human-Robot Interaction. I. INTRODUCTION In human-robot interaction, and particularly in teleoperation, accurately interpreting the user’s intent is crucial. Regardless of the control strategy used, biological signals such as electroencephalography (EEG) can provide valuable insights, particularly through Error-Related Potentials (ErRPs). ErRPs are unconscious brain responses elicited when the teleoperated robot’s action deviates from the user’s expectation [1]. While the use of ErRPs for real-time correction/adaptation in response to remote robot control errors—as well as the challenges associated with their detection—has already been reported in the literature [1]–[4], their relationship with task difficulty and user cognitive load remains largely unexplored. To address this, a systematic protocol for EEG data collection in a virtual environment is designed and proposed, featuring teleoperation and telemanipulation tasks with increasing cognitive load, acquisition of multiple commands from the user, and random injection of intent detection errors into the control chain. A preliminary dataset following the proposed protocol has been recorded involving five participants and is made openly available to the scientific community to support studies on how cognitive load influences the occurrence, amplitude, and timing of these potentials. In the proposed virtual tasks, teleoperation is intentionally simplified (keyboard control of a 2D car and basic grasping with a simulated biological hand) to ensure highly controlled experimental conditions. This simplification is crucial in a field still far from mature development, while the results could impact future works on more realistic teleoperation and telemanipulation scenarios. Authors are with the Department of Electrical, Electronic and Information Engineering (DEI), University of Bologna, Italy (corrisponding author’s email: [email protected]). This work was partially supported by European Commission’s Horizon Europe framework program (project IntelliMan under Grant 10107013). II. SYSTEMATIC DATASET PROTOCOL DESCRIPTION A. Virtual Tasks Design Four tasks were implemented in Unity with custom C# scripts, requiring users to execute keyboard commands based on the displayed scene. Each task was associated with a specific level of cognitive load (CL), increasing in complexity from level 0 to level 3 according to the number of available commands and the inherent difficulty of the task; to eliminate potential order effects, tasks were administered in the following sequence: level 2, level 0, level 3, and level 1. Throughout each task run, trials were presented in a randomized sequence consisting of standard trials—where execution matched the user’s keyboard command—and error trials—where it did not. The sequence was automatically generated at the start of each run and varied between participants. Table I reports the total number of trials for each task run, whereas Fig. 1 illustrates the experimental paradigms for both standard and error trials across tasks. Baseline Task (CL level 0): A yellow car, viewed from above, is positioned at the intersection of four orthogonal streets and can move in four directions: upward, downward, leftward, and rightward. When the car is stationary, the user freely selects the subsequent car direction by pressing the corresponding arrow keys: ↑,↓,←, or →. The background scrolls to simulate car motion while keeping it centered. In standard trials, the car follows the user command, whereas in error trials, it moves differently. Dual-Grasp Task (CL level 1): A simulated hand is centered in the scene, open with its palm facing the user. From outside the scene, a ball moves toward the palm; when still, the user presses ↓to grasp the ball. In standard trials, the hand closes correctly; in error trials, it stays open, requiring the user to repeat ↓in the subsequent trial (either standard or error). After grasping, a bowl enters from outside the scene; when still below the hand, the user presses ↑to open it and drop the ball into the bowl. In standard trials, the hand opens correctly, the ball falls into the bowl, and both slide out of the scene; in error trials, the hand remains closed, requiring the repetition of ↑in the subsequent trial (either standard or error). Tri-Grasp Task (CL level 2): The same hand is positioned as above. From outside, the ball moves toward the palm; when still, the user selects a grasp based on ball’s size and position: power grasp (all fingers closed; ↓) for a large ball centered in the palm, ulnar grasp (only thumb, ring, little; ←) or tripodal grasp (only thumb, index, middle; →) for a small ball on the left/right, respectively. In standard trials, the hand grasps the ball, after which the bowl enters the scene and stops beneath 2025 I-RIM Conference October 17-19, Rome, Italy ISBN: 9788894580570 10.5281/zenodo.17629896 251 TABLE I: Trials repetitions for each task run, considering a minimum of 20 repetitions for each command-trial type combination. Task Commands Trial types N. combinations Total trials per task run Baseline 4: {Up, Down, Left, Right} {Standard, Error}8>160 Dual-grasp 2: {Open, Grasp} {Standard, Error}4 80 Tri-grasp 3: {Power, Ulnar, Tripodal} {Standard, Error}6 120 Five-fingers 5: {Thumb, Index, Middle, Ring, Little} {Standard, Error}10 200 the hand. The user then presses ↑to open the hand and release the ball, which exits the scene together with the bowl. In error trials, the hand closes incorrectly, and the ball escapes. The subsequent trial begins with the ball’s entrance. Five-Fingers Task (CL level 3): The simulated hand is centered in the scene, open with its back facing the user. A colored circle gradually appears beneath the fingertip to flex; then, the user presses 1–5 on the keypad to flex the corresponding finger (thumb to little). In standard trials, the correct finger flexes; in error trials, a different finger does. After flexion, the circle suddenly changes color, the user presses ↑to extend the finger, and the circle gradually disappears. For a clearer understanding of the described tasks, a video demonstration is available here: https://tinyurl.com/4vr3zxdz. B. EEG Data Collection: Experiment Setup Experiments were conducted in a quiet room to minimize external distractions. The setup included a table on which participants rested their arms, a comfortable chair, and a screen and keyboard placed in front of them. A white curtain was used to provide a neutral visual background, and the experimenter was positioned outside the subject’s line of sight. To minimize artifacts in the EEG recordings, participants were instructed to remain relaxed, avoiding body and facial movements, and limiting eye blinks in run sessions. In particular, participants were asked not to look at the keyboard while entering key commands. Before running each task, its procedure was clearly explained to participants, and standard trials were practiced in a preliminary test session to ensure proper understanding. 1.5 s Car motionWait (a) Trials paradigm for the baseline task. 1 s Cue 1.5 s Grasping 1 s Cue 1.5 s OpeningWait 1 s Rest 1 s Cue 1.5 s Open Wait 1 s Cue 1.5 s GraspedWait Wait (b) Trials paradigm for the dual-grasp task. 1 s Cue 1.5 s1 s 1 s OpeningWaitCue Rest 1.5 s Wait Grasping 1 s Wait 1.5 s Grasping 1 s RestCue 1.5 s 1 s Opening RestWait (c) Trials paradigm for the tri-grasp task 1 s Cue 1 s Flexion 1 s 2 s RestOpeningWaitWait (d) Trials paradigm for the five-fingers task. Fig. 1: Task paradigms: Each box is a trial (green: standard, red: error, yellow: both). Wait time is the interval from cue end to key press (black arrow). III. AVAILABLE EEG DATASET Participants and Dataset: A preliminary dataset from five healthy subjects was collected to implement the described systematic protocol. Each participant provided informed consent, confirming their understanding of the experiment. The emotiv_recordings.zip ZIP archive includes a README.pdf file with a detailed description of its contents, a subjects_info.CSV file with anonymized information about involved participants, and a s00x_rec subfolder for each participant. Each subject subfolder contains text files in the format s00x_rec_task_L_eeg.txt and s00x_rec_task_L_triggers.txt, respectively storing the EEG recordings and the custom triggers for each task. The described dataset can be assessed at the following link: https://amsacta.unibo.it/id/eprint/8527. EEG recordings: EEG data were recorded at 256 Hz using the Emotiv EPOC X (©2025 EMOTIV), a wireless headset equipped with 14 electrodes plus 2 references. Both contact quality and EEG signal quality (in %) were repeatedly monitored, and sensor felts were rehydrated as needed. Recordings were acquired exclusively during run sessions using a custom Python script. EEG data samples, along with their local timestamps, were stored in s00x_rec_task_L_eeg.txt. Triggers: Custom triggers were implemented to synchronize user’ commands with the EEG recordings. Each trigger contains information about trials and keyboard inputs along with its local timestamps. In run sessions, triggers were transmitted from C# to Python scripts via TCP (Transmission Control Protocol) and saved in s00x_rec_task_L_triggers.txt. IV. CONCLUSION AND FUTURE WORKS A systematic protocol and a preliminary EEG dataset on ErRPs under varying cognitive load are presented, providing a foundation for future studies on advanced multimodal human–robot interface techniques in robot teleoperation. Future works will focus on (i) expanding the dataset to more subjects, (ii) moving experiments to real-case scenarios replacing the virtual hand with an anthropomorphic robotic hand, and (iii) substituting keyboard commands with muscle-based control. REFERENCES [1] A. Kumar, L. Gao, E. Pirogova, and Q. Fang, “A review of error-related potential-based brain–computer interfaces for motor impaired people,” IEEE Access, 2019. [2] M. Yasemin, A. Cruz, U. J. Nunes, and G. Pires, “Single trial detection of error-related potentials in brain–machine interfaces: a survey and comparison of methods,” Journal of Neural Engineering, 2023. [3] A. F. Salazar-Gomez, J. DelPreto, S. Gil, F. H. Guenther, and D. Rus, “Correcting robot mistakes in real time using eeg signals,” in IEEE ICRA, 2017. [4] A. Fava, A. Lucchese, R. Meattini, G. Palli, V. Villani, and L. Sabattini, “Challenges in detecting and analyzing eeg error-related potentials: Lessons from a case study in hri,” in IEEE ROMAN, 2024. 252