Integrating a hippocampus memory model into a neuromorphic robotic-arm for trajectory navigation

Integrating a hippocampus memory model into a neuromorphic robotic-arm for trajectory navigation Daniel Casanueva-Morato1, Pablo Lopez-Osorio2, Enrique Pi˜ nero-Fuentes1, Juan P. Dominguez-Morales1, Fernando Perez-Pe˜ na2and Alejandro Linares-Barranco1 1Robotics and Technology of Computers Lab., SCORE Lab, Universidad de Sevilla, Sevilla, Spain 2School of Engineering, Universidad de C´ adiz, Spain dcasanue[email protected] Abstract—Neuromorphic engineering endeavors to integrate the computational prowess and efficiency inherent in biological neuronal systems, such as the brain, into contemporary technological systems, primarily through the deployment of spiking neural networks. This research delineates the development and implementation of a bio-inspired sequential hippocampus memory model, which can effectively learn and sequentially recall memories, within a robotic infrastructure. The hippocampus memory model, implemented on the SpiNNaker platform, has been tactically utilized to control a 4-joint event-based robot arm, the ED-ScorBot, by learning and then recalling trajectories via a sequence of memories regarding joint positions. The conveyed spiking information from SpiNNaker is interpreted by an FPGA in real-time to command the event-driven motors of the robotic arm, integrating learned trajectories into physical robotic movement. An empirical exploration validates the model’s capability to govern the robotic arm’s trajectory with precision and dependability while simultaneously demonstrating the potential for incorporating spike-based memory models in robotic applications. This synergistic convergence of neuromorphic engineering and robotics illustrates a viable pathway towards sophisticated, efficient, and adaptable robotic systems capable of learning and reproducing complex tasks, with significant implications for future developments in autonomous robotic applications. I. INTRODUCTION Neuromorphic engineering develops systems that attempt to mimic neuronal systems, such as the brain, in order to incorporate the computational power and efficiency found in nature into current systems [1]. Typically, such systems make use of Spiking Neural Networks (SNNs), which are networks of artificial neurons whose operation and communication are based on the generation and communication of asynchronous electric pulses called spikes [2]. This spike-based information encoding and its processing represent key characteristics of neuromorphic systems. When information is encoded into a spike train at the sensor’s level, the latency required for acquiring a needed amount of information to conclude usable results is considerably reduced [3]. Furthermore, this representation simplifies the consequent operations in managing or processing the information, reducing the power consumption of the circuits implemented for this purpose [4]. This advantage can be maintained not only for feature extraction of sensory systems but also for learning-based methods. One of the fundamental functionalities of any robotic system is the ability to learn and reproduce learned concepts. Within the brain, the hippocampus, with the help of the Entorhinal Cortex (EC), acts as a short-term sequential memory capable of learning and recalling memories quickly by associating different input memories [5]. These characteristics make it an ideal candidate to solve this neuromorphic challenge. Regarding neuromorphic memories, some state-of-the-art works can be found, where the authors propose spike-based memory models, such as [6]–[9], and even sequential alternatives (when recalling, the memory begins to recall not just one memory, but a set of memories sequentially), such as [10], [11]. However, they have low storage capacity, difficulty working with non-orthogonal memories and/or are not sequential. In this work, we proposed a whole robotic infrastructure capable of performing trajectories that are first learned by a hippocampus memory model implemented on SpiNNaker and then recalled due to its sequential architecture. Each of the points in the trajectory for a 4-joint event-based robot arm is sent from the SpiNNaker to an FPGA in real time, which interprets the spiking information and commands the eventdriven motors of the joints. II. MATERIALS AND METHODS A. Hippocampus sequential memory model Fig. 1. Architecture of the spike-based bio-inspired sequential hippocampus memory model, where nis the size of the memory cue. The bio-inspired sequential hippocampus memory model proposed in this work is based on the one presented in [7]. This model has been developed using SNNs and implemented on the SpiNNaker hardware platform [12]. The base model is capable of learning memories, recalling them from a fragment of the memory and forgetting them. By memories, we refer to the information represented by the spiking activity of a set of neurons within a time window. Within memories, the fragment of the memory used in the recall operation (cue) and the rest of the memory (content) can be distinguished. In this work, the base hippocampus model has been extended. A block diagram of the architecture of this new model can be seen in Fig. 1. On the one hand, within the content of the memory, the cue to the next memory in the sequence is added. On the other hand, EC layer 2 (ECl2) and EC layer 5 (ECl5) have been added to create a second recurrent network (with CA3 population being the first). There is evidence that ECl2 has input synapses with the hippocampus and ECl5 has output synapses from the hippocampus, furthermore, both layers are connected to each other in the biological model [13]–[15]. ECl5 presents a set of excitatory synapses that connect the cue of the next memory in the sequence from the output memory to the cue of the input memory in ECl2. Furthermore, ECl5 has a set of inhibitory synapses that connect the cue of the current memory to the cue of the input memory in ECl2. Regarding the number of neurons used, ECl2 and ECl5 populations consist of 26 neurons. 30 neurons are needed for DG, 30 for CA3cue, 21 for CA3cont and 5 for CA1. 16 neurons are set to the output of SpiNNaker (4 neurons per joint), which encode the positions of the joints of the robot. 16 neurons are set as input from the FPGA, which encode the possible trajectories to recall. Thanks to the additions performed to the model presented in [7], the model is able to learn memories that are sequentially related and recall them in sequence. The addition of inhibitory synapses between ECl5 and ECl2 ensures greater stability during the characteristic oscillatory activity that is present during learning and recalling memories. At the operational level, the main difference from the base model is in the recall operation. At the beginning of the recall operation, the cue corresponding to the memory to be recalled is introduced into the network. After the internal oscillations of the model, the model will obtain the complete memory at the output. The part corresponding to the cue of the next memory will return to the input of the network thanks to the ECl5-ECl2 synapses, starting the recall of the next memory in the sequence. This chain of memories will continue until the last memory in the sequence is reached, which next cue is characterized by the absence of neural activity. The frequency of recall operations will be given by the delay at ECl5-ECl2 synapses. For more information on the operations of learning, recalling of single memories and forgetting, see [7]. B. Robotic arm The robotic arm used for this work is a ScorBot-ER VII model [16] that has been modified to work with an Event- Driven (ED) controller [17], giving it the name Event-Driven Fig. 2. Robotic arm diagram (adapted from [16]. ScorBot (ED-Scorbot). This arm has six degrees of freedom (DoF) as represented in Fig. 2. The original controller was completely replaced by a set of FPGA-based infrastructure, allowing the control and power of the motors in a spikebased way, moving from pulse-width-modulation (PWM) to pulse-frequency-modulation (PFM). A Zynq 7100 PSoC from Xilinx hosts the VHDL description of an adapted Spike-based PID (SPID) motor controller [18] for joint position control. Despite this controller, other works have explored the use of SNN for implementing PID controllers on neuromorphic hardware, like [19]–[21] where Loihi was used on different robots. Those FPGA-based controllers can convert the received position reference into a spike train [22] and then use spikebased processing building blocks [23] to design the controller. The embedded computer of the Zynq runs a Petalinux OS that allows remote control of the robot and the autonomous reproducibility of selected trajectories. The predecessors of this robot are the ED-Biorob [24] and the anthropomorphic hand [25]. The ED-Scorbot logic has been improved to allow the monitoring of internal spiking information with the use of an Address-Event-Representation (AER) monitor [26] and to allow the use of AER-based spiking inputs for generating the position references of the joints from external AER systems, like DYNAP-SE2 [27] [28], Loihi [29] or SpiNNaker [12], as in this work, where the hippocampus model is implemented. This model can then be used to learn and afterward elicit an arbitrary trajectory in the arm’s workspace, which can in turn be collected by the ED-Scorbot’s infrastructure. The existing infrastructure facilitates the automated collection of data for trajectories specified by the user in standard mode. In this mode, the trajectory data is transmitted from within the infrastructure logic. Additionally, the infrastructure supports custom data collection in external mode, where the trajectory data is transmitted from an external source. This latter scenario is exemplified in [17]. Fig. 2 depicts a schematic representation of the robotic arm. The data collection method is facilitated by the robot infrastructure, as previously mentioned. The particular implementation entails the utilization of many techniques, primarily the continuous examination of the robot’s joint state and its con- current transmission via an asynchronous protocol (MQTT), in a real-time manner. This feature facilitates the implementation of both automated and customized data collection procedures. C. Hippocampus-robot communication: SpiNNaker-FPGA interface The SpiNNaker platform communicates the address of the spiking neurons using the 2-to-7 encoding protocol with a physical layer at 1.8 v. However, the ED-Scorbot Zynq platform is implemented for receiving 16-bit AER spiking information with a physical layer at 3.3 v. In this work, an HDL has been used to convert one protocol into the other, plus a specifically developed PCB for voltage level conversion and translation of different physical connectors, which was presented in [30]–[32]. Fig. 6 shows the electronics required in this work for interconnecting the SpiNNaker and the ED-Scorbot. The AER monitor allows for collecting SPIDs’ activity while a trajectory is performed. As detailed in Section II-A, the hippocampus model implemented in the SpiNNaker requires a cue to elicit the sequence of joints’ positions learned and stored in the memory. This first cue of a trajectory is generated from the embedded Linux OS through a command shell, which modifies a particular register in the logic that produces an output AER-encoded spike from the ED-Scorbot to the SpiNNaker. This will start the whole process. Every memory elicited from the hippocampus memory model contains not only the next cue but also the content, which represents an address of a neuron that codifies the onevalue bit position of the 4 joints’ target angle. Therefore, SpiNNaker will send as many spikes as ’1’s encoded for the reference of the joints in a consecutive way. The finitestate machine implemented in the logic receives these events for a configurable wait time (5 ms in our experiment) and, when this time has passed, the 4 joints’ referenced angles are commanded to the 4 SPID controllers (see Fig. 3). Fig. 3. Logic flow for the adaptation of the SpiNNaker hippocampus output to the joints references Each joint has been configured to have 16 possible position angle values in their positive range starting at the home TABLE I ED-SCORBOT JOINT’S ANGLES MAPPED FROM SPINNAKER 4-BIT REFERENCE VALUES. Joint 1Joint 2Joint 3Joint 4 4-bit value Angle SR Angle SR Angle SR Angle SR 1 10.4 32 1.70 16 5.13 16 3.81 64 220.8 64 3.40 32 10.26 32 7.63 128 331.2 96 5.10 48 15.40 48 11.44 192 441.6 128 6.80 64 20.54 64 15.25 256 552.0 160 8.50 80 25.68 80 19.07 320 662.4 192 10.20 96 30.82 96 22.88 384 772.8 224 11.90 112 35.96 112 26.69 448 883.2 256 13.60 128 41.10 128 30.51 512 993.6 288 15.30 144 46.24 144 34.32 576 10 104.0 320 17.00 160 51.38 160 38.13 640 11 114.4 352 18.70 176 56.52 176 41.95 704 12 124.8 384 20.40 192 61.66 192 45.76 768 13 135.2 416 22.10 208 66.80 208 49.57 832 14 145.6 448 23.80 224 71.94 224 53.39 896 15 156.0 480 25.50 240 77.08 240 57.20 960 position of the robot. Only 4 bits are needed per joint, thus, 16 bits are used for the 4 joints in this work. Therefore, once all the possible ’1’s or spikes for these 16 bits (corresponding to the output 16-neuron population from SpiNNaker) are received from SpiNNaker as the elicited next point of the trajectory, the commands can be sent to the robot’s SPID joints’ controller. III. RESULTS An experimental study was conducted using the hippocampus memory model that was previously introduced. The experiment involved implementing a trajectory that was directly controlled from the hippocampus model to the actuators of the ED-Scorbot. Subsequently, the data was gathered and subjected to analysis. Fig. 4 shows a graphical representation of the data. The trajectory that was executed can be observed, which entailed a step-by-step motion from the robot’s home location to a far destination and afterward returning. Table I shows the 4-bit reference value understood by SpiNNaker output converted to the robot’s angle space. SR is the spiking reference that internal SPID uses to generate a spiking periodic signal that moves a joint to a particular angle. All details about this robot characterization are available at [17]. Fig. 5 shows a motion picture of the robot while performing the trajectory. This experiment showcases the ability of an external entity to control the robot’s motors while concurrently gathering realtime data via the infrastructure, thus highlighting the potential of the robotic platform. With respect to the hippocampal model, this study demonstrates the capacity of a model, as exemplified in this research, to learn and recall a series of data. In this particular instance, the data was construed as a trajectory of points; however, the model’s ability to assimilate arbitrary input remains evident. IV. CONCLUSION In the recent work presented, the cutting-edge integration of the hippocampal model into the robotic infrastructure is notably evidenced. Through the event-based robotic arm with 4 joints and the SpiNNaker platform (see Figure 6), this study Fig. 4. Joints evolution of the ED-Scorbot for a trajectory elicited from the hippocampus model implemented on SpiNNaker. The trajectory consisted on going from position 0 to position 15 in each of the four joints at the same time in steps of 1, and then returning to position 0 in the same manner. Fig. 5. Motion picture of the robot while performing the 15-point trajectory elicited from the hippocampus model on SpiNNaker. Fig. 6. PCB infrastructure on ED-Scorbot with SpiNNaker connection. showcases how neuromorphic engineering is taking a pivotal role in real-world robotics. By utilizing a memory model that can learn and recall, it has been possible to guide robotic trajectories, paving the way for the execution of complex, memory-informed robotic movements. This approach, as detailed in the paper, not only validates the theoretical foundations of the employed hippocampal model but also suggests a broader range of applications, particularly in navigating and operating within dynamic environments. Following the methodology presented in the study, the robustness and potential efficacy of the neuromorphic model in experimental applications, such as trajectory control in the robotic arm, have been highlighted. This advancement demonstrates that robotic control systems, as detailed in the paper, can transition from being theoretically promising to having tangible practical utility, marking a milestone in the adoption of neuromorphic systems for practical applications. The success of this transition not only reinforces the practicality of the hippocampal model but also lays the groundwork for extending such systems across various robotic applications. The study in question also sheds light on future research directions. Based on the results and the detailed implementation of the hippocampal memory model, a horizon for exploring broader applications of neuromorphic systems across diverse robotic platforms and operational contexts is opened. The combination of reduced power consumption with efficient information processing and sequential memory has the potential to redefine autonomous robotic navigation and learning. This approach, as outlined in the paper, promises to drive significant advancements in emerging fields, such as autonomous vehicles and industrial robotics. ACKNOWLEDGMENT This research was partially supported by project PID2019-105556GB-C33 funded by MCIN/ AEI /10.13039/501100011033. D. C.-M. and E. P.-F. were supported by a ”Formaci´ on de Profesor Universitario” Scholarship from the Spanish Ministry of Education, Culture and Sport. P. L.-O. would like to thank Alejandro Linares- Barranco and his group for hosting them during a 1-month internship between 13th July 2023 and 13th August 2023, during which the bidirectional communication between FPGA and SpiNNaker was achieved. REFERENCES [1] G. Indiveri, B. Linares-Barranco, T. J. Hamilton, A. Van Schaik, R. Etienne-Cummings, T. Delbruck, S.-C. Liu, P. Dudek, P. H¨ afliger, S. Renaud et al., “Neuromorphic silicon neuron circuits,” Frontiers in neuroscience, vol. 5, p. 9202, 2011. [2] Z. Sun, V. Cutsuridis et al., “Brain simulation and spiking neural networks,” Cognitive Computation, pp. 1–3, 2023. [3] E. Forno, V. Fra, R. Pignari, E. Macii, and G. Urgese, “Spike encoding techniques for iot time-varying signals benchmarked on a neuromorphic classification task,” Frontiers in Neuroscience, vol. 16, 2022. [Online]. Available: https://www.frontiersin.org/articles/10.3389/ fnins.2022.999029 [4] C. Farabet, R. Paz, J. Perez-Carrasco, C. Zamarre˜ no, A. Linares- Barranco, Y. LeCun, E. Culurciello, T. Serrano-Gotarredona, and B. Linares-Barranco, “Comparison between frame-constrained fixpixel-value and frame-free spiking-dynamic-pixel convnets for visual processing,” Frontiers in Neuroscience, vol. 6, 2012. [Online]. Available: https://www.frontiersin.org/articles/10.3389/fnins.2012.00032 [5] E. T. Rolls, Brain computations: what and how. Oxford University Press, USA, 2021. [6] D. Casanueva-Morato, A. Ayuso-Martinez, J. P. Dominguez-Morales, A. Jimenez-Fernandez, and G. Jimenez-Moreno, “Spike-based computational models of bio-inspired memories in the hippocampal ca3 region on spinnaker,” in 2022 International Joint Conference on Neural Networks (IJCNN). IEEE, 2022, pp. 1–9. [7] ——, “A bio-inspired implementation of a sparse-learning spike-based hippocampus memory model,” arXiv preprint arXiv:2206.04924, 2022. [8] R. Mueller and A. V. Herz, “Content-addressable memory with spiking neurons,” Physical Review E, vol. 59, no. 3, p. 3330, 1999. [9] M. Matsugu and A. L. Yuille, “Spatiotemporal information storage in a content addressable memory using realistic neurons,” Neural Networks, vol. 7, no. 3, pp. 419–439, 1994. [10] C. H. Tan, E. Y. Cheu et al., “Associative memory model of hippocampus ca3 using spike response neurons,” in International conference on neural information processing. Springer, 2011, pp. 493–500. [11] C. H. Tan, H. Tang, K. C. Tan, and M. Yuan, “A hippocampus ca3 spiking neural network model for storage and retrieval of sequential memory,” in 2013 IEEE Conference on Cybernetics and Intelligent Systems (CIS). IEEE, 2013, pp. 134–139. [12] S. B. Furber, F. Galluppi, S. Temple, and L. A. Plana, “The SpiNNaker project,” Proceedings of the IEEE, vol. 102, no. 5, pp. 652–665, 2014. [13] M. E. Hunt, R. P. Kesner, and R. B. Evans, “Memory for spatial location: Functional dissociation of entorhinal cortex and hippocampus,” Psychobiology, vol. 22, no. 3, pp. 186–194, 1994. [14] C. B. Canto, F. G. Wouterlood, M. P. Witter et al., “What does the anatomical organization of the entorhinal cortex tell us?” Neural plasticity, vol. 2008, 2008. [15] M. P. Witter, T. P. Doan, B. Jacobsen, E. S. Nilssen, and S. Ohara, “Architecture of the entorhinal cortex a review of entorhinal anatomy in rodents with some comparative notes,” Frontiers in systems neuroscience, vol. 11, p. 46, 2017. [16] E. Robotec, “Scorbot er-vii user manual,” Eshed Robotec Limited, Dec. 1998. [Online]. Available: http://intelitekdownloads.com/Manuals/ Robotics/Discontinued%20Machines/ER-VII/ER VII 100017.pdf [17] S. Canas-Moreno, E. Pi˜ nero-Fuentes, A. Rios-Navarro, D. Cascado- Caballero, F. Perez-Pe˜ na, and A. Linares-Barranco, “Towards neuromorphic fpga-based infrastructures for a robotic arm,” Autonomous Robots, pp. 1–15, 2023. [18] A. Jimenez-Fernandez, G. Jimenez-Moreno, A. Linares-Barranco, M. Dominguez-Morales et al., “A neuro-inspired spike-based pid motor controller for multi-motor robots with low cost fpgas,” Sensors, vol. 12, no. 4, pp. 3831–3856, 2012. [19] R. K. Stagsted, A. Vitale, A. Renner, L. B. Larsen, A. L. Christensen, and Y. Sandamirskaya, “Event-based pid controller fully realized in neuromorphic hardware: a one dof study,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020, pp. 10 939– 10 944. [20] S. Stroobants, J. Dupeyroux, and G. De Croon, “Design and implementation of a parsimonious neuromorphic pid for onboard altitude control for mavs using neuromorphic processors,” ser. ICONS ’22. New York, NY, USA: Association for Computing Machinery, 2022. [Online]. Available: https://doi.org/10.1145/3546790.3546799 [21] J. Zhao, N. Risi, M. Monforte, C. Bartolozzi, G. Indiveri, and E. Donati, “Closed-loop spiking control on a neuromorphic processor implemented on the icub,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 10, no. 4, pp. 546–556, 2020. [22] F. Gomez-Rodriguez, R. Paz, L. Miro, A. Linares-Barranco et al., “Two hardware implementations of the exhaustive synthetic aer generation method,” in Computational Intelligence and Bioinspired Systems. Springer Berlin Heidelberg, 2005, pp. 534–540. [23] A. Jimenez- Fernandez, A. Linares-Barranco, R. Paz-Vicente, G. Jimenez, and A. Civit, “Building blocks for spikes signals processing,” in The 2010 International Joint Conference on Neural Networks (IJCNN), July 2010. [24] A. Linares-Barranco, F. Perez-Pe˜ na, A. Jimenez-Fernandez, and E. Chicca, “Ed-biorob: A neuromorphic robotic arm with fpga-based infrastructure for bio-inspired spiking motor controllers,” Frontiers in Neurorobotics, vol. 14, p. 96, 2020. [25] A. Linares-Barranco, R. Paz-Vicente, G. Jimenez, J. L. Pedreno-Molina et al., “Aer neuro-inspired interface to anthropomorphic robotic hand,” in The 2006 IEEE International Joint Conference on Neural Network Proceedings, July 2006, pp. 1497–1504. [26] R. Berner, T. Delbruck, A. Civit-Balcells, and A. Linares-Barranco, “A 5 meps $100 usb2.0 address-event monitor-sequencer interface,” in 2007 IEEE International Symposium on Circuits and Systems, 2007, pp. 2451– 2454. [27] O. Richter, C. Wu, A. M. Whatley, G. K¨ ostinger, C. Nielsen, N. Qiao, and G. Indiveri, “Dynap-se2: a scalable multi-core dynamic neuromorphic asynchronous spiking neural network processor,” 2023. [28] A. Linares-Barranco, E. Pinero-Fuentes, S. Canas-Moreno, A. Rios- Navarro, E. Maryada, J. Z. C. Wu, D. Zendrikov, and G. Indiveri, “Towards hardware implementation of wta for cpg-based control of a spiking robotic arm,” in 2022 IEEE International Symposium on Circuits and Systems, 2022, pp. 1–4. [29] M. Davies, N. Srinivasa, T.-H. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jain et al., “Loihi: A neuromorphic manycore processor with on-chip learning,” Ieee Micro, vol. 38, no. 1, pp. 82–99, 2018. [30] J. P. Dominguez-Morales, A. Rios-Navarro, D. Gutierrez-Galan, R. Tapiador-Morales, A. Jimenez-Fernandez, E. Cerezuela-Escudero, M. Dominguez-Morales, and A. Linares-Barranco, “Live demonstration—multilayer spiking neural network for audio samples classification using spinnaker,” in 2017 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, 2017, pp. 1–1. [31] J. P. Dominguez-Morales, Q. Liu, R. James, D. Gutierrez-Galan, A. Jimenez-Fernandez, S. Davidson, and S. Furber, “Deep spiking neural network model for time-variant signals classification: a real-time speech recognition approach,” in 2018 International Joint Conference on Neural Networks (IJCNN). IEEE, 2018, pp. 1–8. [32] J. P. Dom´ ınguez Morales, “Neuromorphic audio processing through realtime embedded spiking neural networks.” 2018.