Neuromorphic auditory computing: towards a digital, event-based implementation of the hearing sense for robotics

UNIVERSIDAD DE SEVILLA DOCTORAL THESIS Neuromorphic auditory computing: towards a digital, event-based implementation of the hearing sense for robotics Author: Daniel Gutiérrez Galán Supervisors: Dr. Alejandro Linares Barranco Dr. Ángel F. Jiménez Fernández A thesis submitted in fulfillment of the requirements for the degree of Doctor of Philosophy in the Robotics and Computer Technology Lab. Departamento de Arquitectura y Tecnología de Computadores July, 2022 iii Declaration of Authorship I, Daniel Gutiérrez Galán, declare that this thesis, titled “Neuromorphic auditory computing: towards a digital, event-based implementation of the hearing sense for robotics”, and the work presented in it are my own. I confirm that: • This work was done wholly or mainly while in candidature for a research degree at this University. • Where any part of this thesis has previously been submitted for a degree or any other qualification at this University or any other institution, this has been clearly stated. • Where I have consulted the published work of others, this is always clearly attributed. • Where I have quoted from the work of others, the source is always given. With the exception of such quotations, this thesis is entirely my own work. • I have acknowledged all main sources of help. • Where the thesis is based on work done by myself jointly with others, I have made clear exactly what was done by others and what I have contributed myself. Signed: Date: v UNIVERSIDAD DE SEVILLA Abstract Escuela Politécnica Superior Departamento de Arquitectura y Tecnología de Computadores Doctor of Philosophy Neuromorphic auditory computing: towards a digital, event-based implementation of the hearing sense for robotics by Daniel Gutiérrez Galán In this work, it is intended to advance on the development of the neuromorphic audio processing systems in robots through the implementation of an opensource neuromorphic cochlea, event-based models of primary auditory nuclei, and their potential use for real-time robotics applications. First, the main gaps when working with neuromorphic cochleae were identified. Among them, the accessibility and usability of such sensors can be considered as a critical aspect. Silicon cochleae could not be as flexible as desired for some applications. However, FPGA-based sensors can be considered as an alternative for fast prototyping and proof-of-concept applications. Therefore, a software tool was implemented for generating open-source, user-configurable Neuromorphic Auditory Sensor models that can be deployed in any FPGA, removing the aforementioned barriers for the neuromorphic research community. Next, the biological principles of the animals’ auditory system were studied with the aim of continuing the development of the Neuromorphic Auditory Sensor. More specifically, the principles of binaural hearing were deeply studied for implementing event-based models to perform real-time sound source localization tasks. Two different approaches were followed to extract interaural time differences from event-based auditory signals. On the one hand, a digital, event-based design of the Jeffress model was implemented. On the other hand, a novel digital implementation of the Time Difference Encoder model was designed and implemented on FPGA. Finally, three different robotic platforms were used for evaluating the performance of the proposed real-time neuromorphic audio processing architectures. An audio-guided central pattern generator was used to control a vi hexapod robot in real-time using spiking neural networks on SpiNNaker. Then, a sensory integration application was implemented combining sound source localization and obstacle avoidance for autonomous robots navigation. Lastly, the Neuromorphic Auditory Sensor was integrated within the iCub robotic platform, being the first time that an event-based cochlea is used in a humanoid robot. Then, the conclusions obtained are presented and new features and improvements are proposed for future works. vii Acknowledgements Becoming a Doctor of Philosophy has been one of my biggest dreams since I was young. The path has been long and the task has turned difficult, but there have been people all along this time who either have support me or have hated me. To those who have helped me in any kind of situation: thank you all very much. I have no words to let you all know how much I appreciate you. To those who have tried to bring me down: you were my biggest motivation. I have hundreds of words for those people, but only one that can be written here: "empathy". I bought my first computer when I was 9 years old. All my friends asked for the Play Station 2. Instead, I asked for a Pentium 4 with 256 MBytes of RAM memory. Fransico José Junquero told me "this is going to be the future, and you have the potential to learn whatever you want". Without that advice, I would not have asked for a computer, I would not have written these lines. Thank you very much, this thesis also belongs to you. Then, I started to show interest for mathematics. When I was 14 years old, although my marks in math were good, my teacher D. Raúl Espinosa asked me "What degree would you like to study at the University?", and I replied "Computer Engineering". Without any doubt, he told me "Man, your study methodology is shit. You must change it if you want to survive to the first year of university." He was right, and my marks in math during those two years were horrible. Nevertheless, I learned to be self-critical and to identify when I was doing things wrong. Many thanks, Raúl. Right after that period, in the year before moving to the university, I met one of the best teachers I have ever had: Carmelo. He taught me the best methodology to study any science-related subject. Nowadays, I still use his methodology also for this thesis. And because of that, studying math was not a nightmare anymore; instead, it was a game to be enjoyed. I will teach my students by using the same procedure. It will be an honour. From the University of Seville, where I started to study Computer Engineering in 2014, I would like to thanks all the professors I had, because I took the best from all of them. In particular, I would like to thank David Ruíz Cortés, since he was the first who believed in me. And I also have to say sorry, since we started working together in big data but I realized I preferred hardwarerelated tasks. I will never forget our first meeting when you told me "Fight to be the boss and then change every single thing that it is not fair here". I am on my way. In addition, thanks to all my colleagues from the Department of Architecture and Technology of Computers, to which I belong since 2016, for all the advice, both scientist and non-scientist discussions, good and bad moments. You all became my family when I started to spend more than 10 hours per day in the xiv A.1.5 OpenNAS output interface screen . . . . . . . . . . . . . . . 244 A.1.6 OpenNAS destination folder screen . . . . . . . . . . . . . . 245 A.1.7 OpenNAS generation success screen . . . . . . . . . . . . . . 246 B NASIC test PCB 247 B.1 NASICtestPCBfiles ...........................247 B.1.1 Schematic: NASIC_test_pcb.SchDoc ...............248 B.1.2 Board top: NASIC_test_pcb.PcbDoc ...............249 B.1.3 Board bottom: NASIC_test_pcb.PcbDoc .............251 C Generic purpose PCBs 253 C.1 ADC-PDM microphones board PCB files . . . . . . . . . . . . . . . 253 C.1.1 Schematic: ADC_PDM_mic_board.SchDoc ...........254 C.1.2 Board top: ADC_PDM_mic_board.PcbDoc ...........255 C.1.3 Board bottom: ADC_PDM_mic_board.PcbDoc .........255 C.2 ZTEX 2.13 base board PCB files . . . . . . . . . . . . . . . . . . . . . 256 C.2.1 Schematic: ZTEX_base_board.SchDoc ..............257 C.2.2 Board top: ZTEX_base_board.PcbDoc ..............258 C.2.3 Board bottom: ZTEX_base_board.PcbDoc ............259 xv List of Figures 1.1 Cajal’s drawing of neurons in the hippocampus, a region of the brain important to memory. . . . . . . . . . . . . . . . . . . . . . . . 10 1.2 Simplified neuron anatomy. . . . . . . . . . . . . . . . . . . . . . . . 11 1.3 Diagram of a spike generated by a neuron (taken from (Domínguez Morales, 2018)). ...................... 12 1.4 Schematic of pixel circuit from the Mahowald retina model (taken from (Mead, 1990)............................. 14 1.5 SpiNNaker machine project from chip to cabinete (taken from (Sugiarto et al., 2017) ........................... 16 1.6 Transmission of an event using AER representation. Image taken from (Lazzaro et al., 1993)......................... 17 1.7 Screenshot of NAVIS tool. . . . . . . . . . . . . . . . . . . . . . . . . 19 1.8 Main window of NAVIS tool showing the cochleogram of the sentence "En un lugar de La Mancha". Blue dots represent the output of the left 64-channels, while orange dots represent the output for the right channels. Image taken from (Dominguez- Morales et al., 2017c)............................ 19 1.9 Anatomy of the ear. Image taken from (Patton et al., 2012). . . . . . 22 1.10 Detailed cross section of the cochlea. . . . . . . . . . . . . . . . . . . 24 1.11 Inner structure of the Organ of Corti. . . . . . . . . . . . . . . . . . . 25 1.12 Effect of sound waves on cochlear structures. Image taken from (Patton et al., 2012)............................. 26 1.13 Tonotopic distribution of the cochlea (A). Localization of highfrequency (B), medium-frequency (C) and low-frequency (D) responses in the cochlea. Image taken from (Domínguez Morales, 2018)..................................... 27 1.14 Ascending Auditory Pathways scheme. Neural signals can travel from the spiral ganglion to auditory cortex via numerous pathways. Here, a primary pathway is shown schematically (at left) and through brain stem cross sections. Notice that only the connections from one side are illustrated at right image. Image taken from (Bear et al., 2020). ...................... 29 1.15 Cochlear nerve fibers terminate in the dorsal and ventral cochlear nuclei in a tonotopic organization. Image taken from (Kandel et al., 2000)..................................... 31 xvi 1.16 Phase locking in the response of auditory nerve fibers. Sound at a low frequency can elicit a phase-locked response, either (a) on every cycle of the stimulus or (b) on some fraction of the cycles. (c) At high frequencies, the response does not have a fixed-phase relationship to the stimulus. Image taken from (Bear et al., 2020). . 33 1.17 Phase-locked discharges of an auditory nerve fiber (A) and a spherical bushy cell of the Anteroventral Cochlear Nucleus (AVCN) (B). Image taken from (Schnupp et al., 2011). ........ 34 1.18 Schematic drawing of primary auditory sound localization circuits in the mammalian brainstem. Image taken from (Kandler et al., 2009)..................................... 35 1.19 The tapped delay lines, coincidence detectors and coincidence counters (rate integrators) of the Jeffress model. Image taken from (Cariani, 2011)................................ 36 1.20 Sensory and cortical information flows in the Superior Colliculus (SC). The superficial layer of the SC is directly connected to the optic tract, while the deep layers receive auditory (from the inferior colliculus) and somatosensory input. Cortical feedback moderates the processing in the deep layer and influences the motor outputs. Image taken from (Pavlou and Casey, 2010). ............. 38 1.21 Historical tree diagram of different artificial cochleae developed. Image taken from (Domínguez Morales, 2018) ............ 40 1.22 Lyon’s model’s cochlea’s filters’ block diagram. Image taken from (Domínguez Morales, 2018) ....................... 42 1.23 Frequency response of Lyon’s model (64 sections) for the following characteristic frequencies: 3.0, 2.0, 1.0, 0.6 and 0.3 kHz. Image taken from (Miró Amarante, 2013) ................... 42 1.24 Graphical representation of the filterbank and filter-cascade architectures in the Lyon-Katsiamis model. Image taken from (Katsiamis et al., 2007) .......................... 44 1.25 Cutoff frequencies (Hz) distribution in Watts’ artificial cochlea (left) and van Schaik’s (right). Image taken from (Van Schaik et al., 1996) .................................... 46 1.26 Frequency response of a second order filter in a parallel topology (a) and in a cascade topology (b). Image taken from (Domínguez Morales, 2018) ....................... 46 1.27 Block diagram of the digital cochlea proposed by Summerfield et al. Image taken from (Summerfield and Lyon, 1992)......... 48 1.28 Frequency response of the digital cochlea implementation by Leong et al. Image taken from. Image taken from (Leong et al., 2003) .................................... 49 xvii 1.29 (a) Global NAS architecture. (b) Filter banks with Cascade topology, CFB. (c) Single CFB stage containing an SLPF and an SH&F. Image taken from. Image taken from (Jimenez-Fernandez et al., 2017)................................. 51 3.1 Block diagram of the complete architecture of a binaural Neuromorphic Auditory Sensor (NAS). . . . . . . . . . . . . . . . . 58 3.2 Design flow diagram for full NAS synthesis. . . . . . . . . . . . . . 59 3.3 OpenNAS class diagram. . . . . . . . . . . . . . . . . . . . . . . . . . 60 3.4 OpenNAS tool usage flow. . . . . . . . . . . . . . . . . . . . . . . . . 61 3.5 NASIC’s internal architecture block diagram. . . . . . . . . . . . . . 64 3.6 Histograms from pure tones using NAS on Field-Programmable Gate Array (FPGA). Middle frequencies associated to each frequency channel were: channel 0, 22000 Hz; channel 1, 8090 Hz; channel 2, 2975 Hz; channel 3, 1094 Hz; channel 4, 402 Hz; channel 5, 148 Hz; channel 6, 54 Hz; and channel 7, 20 Hz. . . . . . . . . . . 65 3.7 Sonograms from pure tones using NAS on FPGA. . . . . . . . . . . 67 3.8 ASIC design workflow. Image taken from https://www. einfochips.com/ ............................. 68 3.9 NASIC cells distribution after the place & route process. . . . . . . 69 3.10NASIClayout................................ 70 3.11 NASIC test Printed Circuit Board (PCB) block diagram. . . . . . . . 72 3.12 NASIC test PCB with the audio input board. . . . . . . . . . . . . . 73 3.13 Histograms from pure tones using NASIC. . . . . . . . . . . . . . . 74 3.14 Sonograms from pure tones using NASIC. . . . . . . . . . . . . . . . 75 3.15 Comparison between FPGA and chip 3. . . . . . . . . . . . . . . . . 77 4.1 Superior Olivary Complex in biology. Green lines represents excitatory connections, while red lines indicates inhibitory connections. Image taken from (Liu et al., 2013). ........... 80 4.2 Block diagram of the NSSOC model for FPGA. . . . . . . . . . . . . 81 4.3 Representation of the phase lock effect. Image taken from (Liu et al., 2013).................................. 83 4.4 Example of the phase lock effect. . . . . . . . . . . . . . . . . . . . . 83 4.5 Block diagram and simulation results of the spherical bushy cell model. ................................... 84 4.6 Histograms and sonograms for a 500 Hz pure tone with 0.5 and 1.5 of amplitude using a NAS without phase-lock. . . . . . . . . . . . . 85 4.7 Histograms and sonograms for a 500 Hz pure tone with 0.5 and 1.5 of amplitude using a NAS with the phase-lock module. . . . . . . . 86 4.8 Examples of recordings using NAS without and with phase-lock. . 87 4.9 Block diagram of the proposed implementation for the Jeffress model. ................................... 89 4.10 Finite State Machine (FSM) of the Spike Hold&Coincidence Fire (SHCF)module............................... 90 xviii 4.11 Simulation results of the SHCF module. States 0, 2, 3, and 4 corresponds to IDLE,WAIT_RIGHT,COINCIDENCE, and REFRACTORY,respectively........................ 91 4.12 FSM of the Delay Line module. . . . . . . . . . . . . . . . . . . . . . 92 4.13 Simulation results of the Delay Line (DL) module. States 0, 1, and 2 to IDLE,HOLD, and FIRE, respectively. . . . . . . . . . . . . . . . 92 4.14 Example of the ITD network when using a sweep. . . . . . . . . . . 94 4.15 Block diagram of the proposed MSO model. . . . . . . . . . . . . . 97 4.16 Block diagram of the MSO local events monitor. . . . . . . . . . . . 98 4.17 Block diagram of the MSO global events monitor. . . . . . . . . . . 99 4.18 Results from a behavioral simulation of a Medial Superior Olive (MSO) model when using a 1000 Hz pure tone, placed in front of the reference, with a distance of 1.5 meters. The plots were generated using pyNAVIS. . . . . . . . . . . . . . . . . . . . . . . . . 101 4.19 Interaural Level Difference (ILD) (encoded as an event-rate) measured over time obtained from the input data that generates the left and right NAS when the sound source is placed at 45° from the head. Figure taken from (Cerezuela-Escudero et al., 2018). . . . 103 4.20 Block diagram of the proposed SOC module. . . . . . . . . . . . . . 105 4.21 Block diagram for the integration between the NAS model and the SOCmodel. ................................106 4.22 Event package format. . . . . . . . . . . . . . . . . . . . . . . . . . . 106 4.23 Virtual room created for testing the Neuromorphic Auditory Complex (NAC). Diamond shape indicates the microphones pair, where blue diamond correspond to the left microphone and red diamond correspond to the right microphone. Therefore, sound sources are placed in front of the microphones pair. The sound source 0 is the one placed completely on the right. . . . . . . . . . . 109 4.24 Comparison between different configurations of the MSO. . . . . . 111 4.25 Frequency comparison using model 2. . . . . . . . . . . . . . . . . . 114 4.26 Concept of cone of confusion. Figure taken from (Risoud et al., 2018).115 4.27 Effect of the cone of confusion for model 2. . . . . . . . . . . . . . . 117 4.28 Theoretical behavior representation of theTime Difference Encoder (TDE) model based on the model proposed by Milde et al.(Milde et al., 2018). a) TDE schematic with facilitatory (fac) and trigger (trig) input and spiking output. b) Case one: A small positive time difference between facilitatory and trigger spikes leads to a high number of output spikes (out). c) Case two: large positive time difference leads to no output spikes. d) Case three: A negative time difference leads to no output spikes. e) Number of TDE output spikes in dependency of time difference ∆tbetween two input events (gain: gain factor, epsc: exponential postsynaptic current, mem: membrane potential). . . . . . . . . . . . . . . . . . . . . . . . 121 xix 4.29 Detailed block diagram of the TDE digital architecture. It is composed of three main blocks: gain-generator block (red), Excitatory Post-Synaptic Current (EPSC)-generator block (blue) and spike-generator block (green). Synchronous modules are indicated by squared corner blocks with a small triangle, while asynchronous modules are indicated with rounded corner blocks. The spike arrow is a 1-bit width signal, where events are either received as input or sent as output. Control arrows are also 1-bit width, and they act as flags. Finally, data arrows can be either n-bit or m-bit width, being used for the internal communication between blocks and also for loading configuration values. . . . . . 123 4.30 Gain-generator block output example. First, the block receives a single facilitatory event. Immediately after, timer_0 is loaded with the "detection_time" signal’s value. The output value of timer_0 decreases by one unit for each time reference tick, which was set to microseconds. Since the "weight"’s value was set to 1 (meaning that the timer’s value is left shifted by one position), the shift_1’s output value is twice the value of timer_0. Then, the block receives multiple facilitatory events in order to show the accumulative behavior. add_0 plot shows the value that is used as input for the spike-generator block. In this case, its value matches the shift_1’s value, since no trigger event was received. . . . . . . . . . . . . . . . 126 4.31 EPSC-generator block output example. First, the model is stimulated with a single facilitatory event before the first trigger event. Then, the current value of shift_1 is loaded in timer_1. The accumulative effect is also shown when multiple triggers are received. The sub_0 module generates an increasing signal, which is used as the clock divider value for the spike generator block. . . 129 4.32 Exhaustive Unsigned Synthetic Spikes Generator (EU-SSG) block diagram. A complete description of both the implementation and the behavior of this module is presented in (Jimenez-Fernandez et al., 2010)..................................130 4.33 An operation example of the proposed model. The detection time was set to 700 µs; the tau value was set to 0; the weight value was set to 4; and the decay value was set to 2. The main clock was set to 50 MHz, the time reference tick was set to 1 MHz, and the data width was set to 16. The ∆tvalue between the facilitatory event and the first trigger event is 200 µs, while the ∆tvalue between the facilitatory event and the second trigger event is 400 µs........133 4.34 Register-Transfer Level (RTL) simulation for twelve basic cases of the TDE model with a time resolution of microseconds. Red, blue, and green are related to facilitatory, trigger, and spike generator, respectively. ................................135 xx 4.35 TDE Interspike Interval (ISI) response for a facilitatory-trigger pair with different ∆t values for a microseconds resolution configuration. Note that the smallest ∆t value used was not zero (no output events would be produced) but 20 ns (one clock cycle). . 136 4.36 TDE ISI response for a facilitatory-trigger pair with different ∆t for a milliseconds resolution configuration with a detection time of 70 ms. .....................................137 4.37 Number of TDE output spikes over ∆t variation for microsecond timereferencetick. ............................138 4.38 Number of TDE output spikes over ∆t variation for millisecond timereferencetick. ............................138 4.39 Individual TDE tuning curves and population tuning curve for microseconds time reference tick. . . . . . . . . . . . . . . . . . . . . 140 4.40 Individual TDE tuning curves and population tuning curve for microseconds time reference tick. . . . . . . . . . . . . . . . . . . . . 140 4.41 Block diagram of the setup for real-time measurements acquisition. 141 4.42 Output spikes captured by using an oscilloscope. . . . . . . . . . . . 142 4.43 Comparing the TDE response from different measurement sources. 143 4.44 Detailed block diagram of the FPGA top module for the proof-of- concept, containing both the NAS and the TDE populations. . . . . 145 4.45 Raster plot of the output events from the TDE population and normalized overall activity. The plots were generated using pyNAVIS tool (Dominguez-Morales et al., 2021b). TDEs 0 to 3 (left population) correspond to indexes 0 to 3 in 4.8, and TDEs 4 to 7 (right population) also correspond to indexes 0 to 3 in the same table. Therefore, TDE 0 and TDE 4 use the same configuration and soon.....................................146 5.1 An octopus-inspired soft robotic arm. Credit: Harvard SEAS. . . . 151 5.2 Block diagram of the entire system. It is composed of the SpiNNaker board, an FPGA-based board, and a 3D-printed hexapodrobotframe............................155 5.3 A) Biological representation of an arthropod’s leg anatomy. B) Hexapod robot leg actuator IDs. . . . . . . . . . . . . . . . . . . . . . 156 5.4 SpiNN-3 machine and ZTEX 2.13 board. . . . . . . . . . . . . . . . . 158 5.5 Diagram of the spiking neural network model used (top) with an in-depth view of the CPG architecture (bottom). . . . . . . . . . . . 159 5.6 NeuroPod FPGA top module overview. . . . . . . . . . . . . . . . . 161 5.7 Output spikes for each gait pattern simulated on SpiNNaker. . . . . 163 5.8 Output spikes from the SpiNNaker simulations that show the gait pattern change behavior. . . . . . . . . . . . . . . . . . . . . . . . . . 164 5.9 Block diagram of the updated NeuroPod system. It is composed of the SpiNNaker board, implementing different networks, an FPGA- based board, in which the NAS was deployed, and a 3D-printed hexapodrobotframe............................168 xxi 5.10 Audio-guided NeuroPod FPGA top module overview. . . . . . . . 169 5.11 Diagram of the complete spiking neural network model used for the audio-guided NeuroPod. This network is composed of several sub-networks, where each of them has a specific task. . . . . . . . . 170 5.12 Audio-guided NeuroPod hardware setup. . . . . . . . . . . . . . . . 171 5.13 Output spikes from the SpiNNaker simulations that show the gait pattern change behavior according to the input auditory stimuli. . 172 5.14 The autonomous robot navigates through a cluttered environment. It tries to reach a sound source while avoiding obstacles. . . . . . . 174 5.15 Complete network. The Optical Flow Encoder (OFE) population consists of the SPatio-Temporal Correlation (SPTC) Leaky Integrate-And-Fire (LIF) population and the spiking Elementary Motion Detector (sEMD) population. The Sound Source Direction (SSD) population includes coincidence detector neurons and an additional hard Winner-Take-All (WTA) network. Each WTA consists of an excitatory LIF neuron population and one Global Inhibitory (GI) neuron. All excitatory LIF neurons are connected to the GI neuron. The GI neuron projects back onto the excitatory LIFneurons.................................177 5.16 embedded Dynamic Vision Sensor (eDVS) and NAS feeding spikes into a SpiNN-3 board through the SpiNNlink connectors. The eDVS can directly process visual input while the NAS receives stereo audio input through an audio jack. . . . . . . . . . . . . . . . 178 5.17 (a) Sensory Integration (SI) network’s heading direction response to a 180 degrees sound source direction sweep. (b) Setup to record Optical Flow (OF) data used in Figure 5.17c. An eDVS is mounted on top of a robotic platform which drives purely translational with a speed of ~0.8 m/s through the scene. The first obstacle is located in the middle of the visual field, the second one on the right side and the last one on the left side. All obstacles are positioned at least 40 cm above the ground so that the robot can drive underneath them. Heading directions for a centered sound source with OF (c) and without OF (d). (e) Heading direction mean and heading direction standard deviation (stdd) with OF (red) and without OF (blue) for five different time periods with different visual scenarios 179 5.18 Hardware setup for the test scenario with real-time input and static sensors’ position. The screen shows the response of the system. Red dot: heading direction; blue dots: optical flow. . . . . . . . . . . 181 5.19 Output spikes from both the heading direction network and optical flow estimation network for the static hardware setup realtimetest. ..................................181 xxii 5.20 Hardware setup for the test scenario with real-time input and dynamic sensors’ position. It is composed by the Neuromorphic Pan-Tilt-Unit including the Event-Based Camera, Neuromorphic Auditory Sensor with 3DIO Binaural Microphone, and SpiNNaker board.....................................182 5.21 The iCub robot. Picture taken from (Parmiggiani et al., 2012). . . . . 184 5.22 (A) Photograph of the iCub head electronics. (B) Photograph of the iCub head electronics. Pictures taken from (Parmiggiani et al., 2012).186 5.23 CAD view of the iCub head components with a block diagram of the proposed NAC-iCub integration. CAD picture taken from (Parmiggiani et al., 2012). ........................188 5.24 Block diagram of the NAC model integrated within the iCub robot. It is based on the one shown in Fig. 4.21, adding just the Address Event Representation (AER)-Head Processing Unit (HPU) wrappermodule. .............................189 5.25 AER-HPU wrapper output event package format. . . . . . . . . . . 189 5.26 Yet Another Robot Platform (YARP) modules diagram for the basic cochlea visualizer application shown in yarpmanager. . . . . . . . . 192 5.27 Cochlea visualizer window showing iCub’s auditory events activity. (A) Events’ activity when no sound is played. Those events correspond to the noise produced by the iCub’s fan. (B) Events’ activity when a 500 Hz tone is played. . . . . . . . . . . . . 193 5.28 Example recording from a woman reading a list of words in front of the iCub robot, with a distance of 0.5 meters. (A) Spikegram. (B) Sonogram. (C) Histogram. (D) Average activity. . . . . . . . . . . . 194 5.29 YARP modules diagram for the sound source localization application shown in yarpmanager. . . . . . . . . . . . . . . . . . . . 196 5.30 YARP visualizers for the auditory perception model. . . . . . . . . . 197 5.31 Overview diagram of the Spiking Neural Networks (SNNs) implemented on SpiNNaker to implement the auditory perception oniCub. ..................................199 5.32 Spiking neural network diagram of the inferior colliculus model implemented on SpiNNaker. . . . . . . . . . . . . . . . . . . . . . . 200 5.33 NAS response plots for a 523 Hz pure tone placed at the left. (A) Spikegram. (B) Sonogram. (C) Histogram. (D) Average activity. . . 202 5.34 MSO response plots for a 523 Hz pure tone placed at the left. (A) MSO spikegram. (B) MSO histogram. (C) MSO localization estimation. (D) MSO heatmap. . . . . . . . . . . . . . . . . . . . . . 203 5.35 NAS’ response plots (Figs. (A) Average activity, (B) Sonogram, (C) Difference, and (D) Histogram) and MSO response plots (Figs. (E) MSO spikegram and (F) MSO localization estimation) for a 523 Hz pure tone while moving the iCub head from left to right and back with a constant velocity. . . . . . . . . . . . . . . . . . . . . . . . . . 207 xxiii 5.36 Screenshot of Gazebo simulator while performing a sound source localization simulation with a virtual neuromorphic iCub and real recordings as input stimuli. . . . . . . . . . . . . . . . . . . . . . . . 208 5.37 Output spikes for each gait pattern simulated on SpiNNaker. . . . . 209 A.1 OpenNAS tool welcome message. . . . . . . . . . . . . . . . . . . . 240 A.2 OpenNAS screen for step 1: common settings. . . . . . . . . . . . . 241 A.3 OpenNAS screen for step 2: audio input interface. . . . . . . . . . . 242 A.4 OpenNAS screen for step 3: audio processing architecture. . . . . . 243 A.5 OpenNAS screen for step 4: spiking output interface. . . . . . . . . 244 A.6 OpenNAS screen for step 5: destination folder selection. . . . . . . 245 A.7 OpenNAS final screen: code generator statistics. . . . . . . . . . . . 246 B.1 NASIC test PCB schematic. . . . . . . . . . . . . . . . . . . . . . . . 248 B.2 NASIC test PCB board top view. . . . . . . . . . . . . . . . . . . . . 249 B.3 NASIC test PCB board top 3D view. . . . . . . . . . . . . . . . . . . 250 B.4 NASIC test PCB board bottom view. . . . . . . . . . . . . . . . . . . 251 B.5 NASIC test PCB board bottom 3D view. . . . . . . . . . . . . . . . . 252 C.1 ADC-PDM microphones board PCB schematic. . . . . . . . . . . . . 254 C.2 ADC-PDM microphones board PCB board top view. . . . . . . . . . 255 C.3 ADC-PDM microphones board PCB board top 3D view. . . . . . . . 255 C.4 ADC-PDM microphones board PCB board bottom view. . . . . . . 255 C.5 ADC-PDM microphones board PCB board bottom 3D view. . . . . 255 C.6 ZTEX 2.13 base board PCB schematic. . . . . . . . . . . . . . . . . . 257 C.7 ZTEX 2.13 base board top view. . . . . . . . . . . . . . . . . . . . . . 258 C.8 ZTEX 2.13 base board bottom view. . . . . . . . . . . . . . . . . . . . 259 C.9 ZTEX 2.13 base board bottom 3D view. . . . . . . . . . . . . . . . . . 260 xxxi Dedicated to all the people who have been mentioned in the acknowledge. 1 Part I Thesis 3 Chapter 1 Introduction “To understand reality, you have to understand how things work. If you do that, you can start to do engineering with it, build things. And if you can’t, whatever you’re doing probably isn’t good science.” – Carver Mead Artificial visual processing is a current hot topic in the technological revolution that we are living in recent years. We have autonomous cars that are able to identify, understand and follow traffic signals, safely drive on a highway, and predict accidents even before the driver. Computer-based visual processing techniques are also used in factories to detect and discard faulty products in the assembly line. The advance is so impressive that even those techniques are being applied in medical fields for detecting diseases with more accuracy than specialized doctors. The sense of vision provides humans with a large quantity of details that help us to understand our environment. This is why the vision could be considered the most important sense. Nevertheless, sometimes visual stimuli need to be put in context to fully understand reality. Among the whole set of human senses, the hearing sense is closely linked to the sight sense. Speech allows us to communicate, interact and share ideas and feelings. It also helps us to make our daily task easier thanks to the voice assistants. The wide range of technologies employed in the development of such devices mixes speech recognition, natural language processing, and contextualized information search, among others. Commonly, the sound is sent to the cloud, where it is processed by high-performance servers applying digital signal processing techniques. Hence, we could say that the processing engine is centralized outside the device. Although this approach has been proved to work, it faces some inconveniences which could be desirable to improve or remove, as power consumption during the "always-on" mode even if there is no data to process, the complexity of algorithms or noise robustness. Researchers have tried to solve 4Chapter 1. Introduction those problems by applying many different techniques until they started to take inspiration from biology. Species have been evolved over millions of years, slowly improving their body shapes, abilities and senses. In this process, the brain has played a key role by adapting itself according to the needs, optimizing task resolution while keeping a low power consumption. Since we could confirm that the brain is one of the most efficient computers ever made, it is the perfect candidate to be mimicked. Humans are able to hear thanks to the ear, which receives sound waves, stimulating the inner ear, and sending the neuronal activity to the auditory cortex through a pathway. Differently to the cloud-based devices, sound’s features are extracted in different brain nuclei across the auditory ascending pathway, being processed by the auditory cortex afterwards. The function carried out by each nucleus is generally specific, which does not mean that the process to do it is simple. And this thought could lead to follow phenomenological modelling of bio-inspired electronics devices instead of an emulation approach. This way, many works can be found in the literature which aim to design, part by part, a human-like hearing sense, starting from the ear and its cochlea, continuing by the extraction of binaural cues for the spatial context in the early brain nuclei, and finishing with sound detection and speech recognition tasks in the auditory cortex. Despite those efforts, the use of a bio-inspired solution for solving auditory tasks is not as high as desired. Maybe because there is a gap between used technologies or approaches; maybe because the sound itself is not enough to take a decision and it needs to be exploited together with other sensory data. But what is true is that the hearing sense is involved in all our daily tasks, from learning to teaching, from protecting to being safe. Therefore, to understand how it works and for applying its benefits to society, either directly by developing bio-inspired cochlear implants or indirectly by mean of homeassistant humanoids robots, this thesis has been carried out with dedication, excitement, and effort. 1.1 Motivation Currently, auditory perception is becoming popular in different research areas. For instance, cochlear implants are considered as one of the best achievements in the bio-engineering field (Zeng et al., 2008; Wilson et al., 1991; Eshraghi et al., 2012). The performance of such devices, inspired in a certain way from biology, is incredibly high, making the life of impaired people better. The technology under the cochlear implants is based on an external digital signal processor connected to an array of electrodes, which is placed in the human cochlea. Nevertheless, the fact that the user needs to have an external device could be a problem in some 1.1. Motivation 5 daily aspects. With the new technologies, such as the neuromorphic engineering, new bio-inspired approaches for auditory sensors could be used for improving the current version of cochlear implants. And it could lead to the creation of a new generation of fully neuromorphic cochlear implants with better features. Although all the efforts were put on cochlear models, there could be other cases in which the disease affect not directly the cochlea but also the following sections of the auditory ascending pathway. This means that the person could have problems for spatial sound source localization, which is an essential task in safety and socialization. Furthermore, auditory perception is closely linked to visual perception, being both of them mixed in the brain. As examples, visual information helps the auditory perception when a person is talking by reading its lips, and the auditory information helps the visual perception to focus the attention by localizing the sound in the space (Massaro and Stork, 1998; Molholm and Foxe, 2005; Khacef et al., 2020). This neuromorphic approach of the hearing sense, besides to help humans, can be also applied to robotics. On the one hand, the low-latency nature of the neuromorphic systems could be exploited for autonomous navigation, using in combination both visual and auditory sensors. This way, an autonomous car would be able to detect moving objects that could be out of its field of view thanks to the sound, reacting to that event in the order of hundreds of microseconds. On the other hand, human-like robots are considered the best platform to test this new bio-inspired systems before moving to real experiments with humans. Their use helps to finely improve the hardware, the software and the final applications by comparing their behavior with the expected human behavior. After having a deep look at the state-of-the-art, and after being identified several aspects related to the neuromorphic auditory perception which can be improved, this work have been motivated to contribute in the following three main blocks: • Facilitate the access of a neuromorphic auditory sensor to the neuromorphic research community: even though there exist different options, like insilico analog cochlea models and some others digital implementations, they are not so accessible due to whether their high price, low availability, or difficulty of being configured and used. For further improvements in the neuromorphic auditory perception field, an open source auditory sensor would be desirable that could offer the researchers a global platform for performing experiments. • Continue with the development of our neuromorphic auditory sensor by adding new useful features: the cochlea model is just the first step in the auditory ascending pathway. Therefore, our final desire would be to have a complete digital, spike-based hearing sense starting from the cochlea up to the auditory cortex. In this path, the next step is the implementation of the brain nucleus where the binaural cues for the sound source localization 6Chapter 1. Introduction are extracted. In addition, current cochlea models does not implement this feature directly. Instead, software models have been developed as a complement with specific requirements, making it difficult to integrate the whole system in the same robotic platform. • Integrate the auditory model within robotic platforms for providing them with the sense of hearing: related to the first item, it is not common to find neuromorphic cochlea models, either hardware or software, already integrated in robots, thus making it difficult to advance in the filed of realtime audio-guided robotics applications. The cochlea models are often used for generating event-based audio datasets or testing local neuromorphic audio applications. But it would be desirable to have a robotic platform that can provide both a neuromorphic sensor set (such as neuromorphic retinas, cochleas, motor control, etc) together with a unique software framework that allows the user to manage the input sensory events, avoiding to have multiple individual sensors managed by multiple individual software. The proposed improvements and contributions presented in this thesis, that have been carried out with the best of my intentions, modesty, effort, and without the aim to have the absolute truth, can be summarized as: • The implementation of an open source tool for automatically generating the design files of a technology-independent, digital neuromorphic cochlea model. Using the NAS as reference (developed by Ángel Jiménez Fernández in his thesis), this software tool is able to generate custom Hardware Description Language (HDL) files ready to be synthetized by FPGA’s oriented tools without the need of being modified by the user. This way, it would not be necessary to have previous knowledge of HDL code. As advantage, the generated HDL code is platform independent, therefore it can be deployed in any FPGA from different vendors. In addition, the first open hardware NAS Application-Specific Integrated Circuit (ASIC) has been designed and fabricated to test the performance of our approach. • The design, implementation, and test of a digital, spike-based model of the human Superior Olivary Complex (SOC) for performing sound source localization tasks. This model takes the output spiking information generated by the NAS as input, and is able to extract useful information about the localization of sounds in the space. Since it can be deployed into an FPGA, this model is also technology-independent, configurable and scalable. In addition, a new approach for the Interaural Time Difference (ITD) estimation has been proposed by using the concept of the Time Difference Encoder (TDE) neuron model as an alternative to the Jeffress model. This TDE model, as well as a proof-of-concept of its usage for the sound source lateralization task, have been designed and implemented in this work. 1.2. Neuromorphic engineering 7 • The integration of the neuromorphic auditory sensor within robotic platforms and the demonstration of real-time neuromorphic audio processing applications. In particular, the NAS has been used in two application-specific robots and in one general purpose robot. First, the NAS was used as input sensor for modifying the movement pattern of an hexapod robot in real-time, where an internal Spiking Central Pattern Generator (SCPG) (implemented on SpiNNaker) was commanded by the input auditory stimuli. Second, the sound source localization information was combined with visual information in order to implement a realtime, audio-visual collision avoidance system. Third, both the NAS and the localization model were integrated within the iCub robot and its control framework for multi-modal sensory processing tasks and auditory perception algorithms development. This thesis is part of the research career of the Robotics and Computer Technology group (RTC, TEP-108), to which the author belongs. This work is focused on and aligned with different tasks that are part of national research projects, which have served as funding, along with the Spanish Ministry of Education, Culture and Sports, thanks to a Formación del Personal Investigador scholarship. The knowledge, materials, code, and documents generated in this thesis have been made open-source (as far as possible) to be used by the research community due to it was funded with public money. These projects are: • COFNET project: Sistema Cognitivo de Fusión Sensorial de Visión y Audio por Eventos (TEC2016-77785-P). • NPP project: Neuromorphic Processor (P051-15/E03). • NPP2 project: Neuromorphic Processor Project Phase 2 (P062-18/E03). • MIND-ROB project: Percepción y Cognición Neuromórfica para Actuación Robótica de Alta Velocidad (PID2019-105556GB-C33). 1.2 Neuromorphic engineering Moderns computers are based on the von Neumann architecture, which was proposed in 1945 by the computer scientist John von Neumann, and it is still being used nowadays. Over the years, most of the efforts from traditional computing companies have been focused on improving the speed of computation without taking too much care of the power consumption. However, the main problem resides on there is a mismatch between the speed of the memory access and the speed of the Central Processing Units (CPUs), leading to the so-called von Neumann bottleneck (Alex, 2009). Furthermore, Moore’s law is thought to come to an end in the next years due to physical constraints such as increasing thermal noise (Kish, 2002). These developments have triggered increasing research efforts for the design of 14 Chapter 1. Introduction FIGURE 1.4: Schematic of pixel circuit from the Mahowald retina model (taken from (Mead, 1990) 30 years either using biophysical or phenomenological approaches (Bartolozzi and Indiveri, 2007; Brette and Gerstner, 2005; Milde et al., 2018). One of the most popular and implemented model is the LIF neuron (Liu et al., 2002; Indiveri et al., 2011a), that basically integrates the input current into the neuron’s membrane potential until it reaches a certain threshold, producing then an output spike. In addition, this model takes into account the leak of the membrane potential voltage over time, reflecting the diffusion of ions through the membrane. This kind of circuits generally consist of tens of transistors and a couple of capacitors, and those transistors can work at either sub-threshold or abovethreshold, thus becoming very low-power consumption devices. These features have allowed to design analog neuromorphic procesors, as DYNAP(Moradi et al., 2017), and analog neuromorphic processing machines, as BrainScaleS (Schemmel et al., 2010). Moreover, researchers have turned bugs into features, as the device mismatch of analog circuits, by using the mismatch’s randomness to increase the robustness of the neuron’s learning procedure. Nevertheless, digital implementations have been also carried out by following different approaches. On the one hand, the neuron’s model can be implemented as a digital circuit to be later deployed into an FPGA or integrated into an ASIC. These circuits, in turn, can implement single or multiple neurons by following either biophysical (Cassidy and Andreou, 2008; Perez-Peña et al., 2019; Tapiador-Morales et al., 2018) or phenomenological (Frenkel et al., 2017; Indiveri et al., 2011b) modelling techniques. In addition, these neuron models can be used as processing unit of a neuromorphic processor, creating thus a Network-on- Chip (NoC) and allowing to implement large neural networks. MorphIC (Frenkel et al., 2019), Loihi (Davies et al., 2018), and SpiNNaker (Furber et al., 2013b) are 1.2. Neuromorphic engineering 15 good examples of massively multicore neuromorphic processors, among others. On the other hand, neuron’s models can be simulated by software algorithms which can be run in parallel computers or Graphical Processing Units (GPUs), taking advantage from their inherent parallel architecture (Knight and Nowotny, 2019; Knight and Nowotny, 2021). This way, large-scale population of neurons can be simulated, somethings even obtaining better performance when compared with neuromorphic hardware solutions, like SpiNNaker (Knight and Nowotny, 2018). SpiNNaker (Spiking Neural Network Architecture) is a massively parallel multicore computing system to model very large SNNs in real time, optimized for neuromorphic applications. Both the system architecture and the design of the SpiNNaker chip have been developed by the Advanced Processor Technologies Research Group (APT) at the University of Manchester. Each SpiNNaker chip consists of eighteen 200 MHz general-purpose ARM968 cores, each one with its own memory. The chip contains a Globally Asynchronous Locally Synchronous (GALS) architecture with an asynchronous packet switching network that is highly optimized for neuromorphic applications (Plana et al., 2007). The communication between them is done via packets carried by a custom interconnect fabric. The transmission of these packets is managed entirely by hardware, giving the overall engine and extremely high bisection bandwidth. It is important to mention that one of the 18 ARM processors is used for management, and another ARM core is reserved. Therefore, only 16 ARM cores are involved in the neuromorphic process. Fig. 1.5 bottom right shows a picture of the SpiNN-5 machine, which has 48 SpiNNaker chips (86 ARM processor cores) (shown in Fig. 1.5 bottom left) and 3 Spartan-6 FPGAs to communicate to other boards. Both boards have a 100 Mbps Ethernet connection that is used as control and I/O interface between the computer and the SpiNNaker board. SpiNN-5 has been used to build a million core massively parallel computer for human brain simulation (Furber and Brown, 2009), shown in Fig. 1.5 top. In general, neuromorphic systems are composed of a neuromorphic sensor (or a group of them) and a set of spiking neural networks with multiple layers. These networks process the information provided by the sensors in the same way that the brain does. However, due to the brain has a high density in terms of neuronal connections, where each of the 105/mm3neurons could be connected to other 10 thousand generating a density of connections up to 4km/mm3(Braitenberg and Schüz, 1998), it is a hard task to design and fabricate this structure in a VLSI system. Nevertheless, a neuron presents a firing rate that ranges between 1-10 Hz, scaling up to kHz or MHz if many hundreds are combined together, meaning that state-of-the-art electronic circuits are much faster than biological neurons. 16 Chapter 1. Introduction FIGURE 1.5: SpiNNaker machine project from chip to cabinete (taken from (Sugiarto et al., 2017) A mechanism that multiplexes the information in time for a set of neurons in a single communication channel was proposed for the first time in 1991 (Sivilotti, 1991; Lazzaro et al., 1993; Lazzaro and Wawrzynek, 1995; Boahen, 2000) with the aim of solving connectivity problems, and it is known as AER. Based on this time difference and the high bandwidth capacity of the VLSI systems, each neuron is identified with a unique address, which is sent through a shared bus whenever the neuron fires a spike. It is an event-driven communication protocol used originally for transferring spikes (action potentials) between neurons in VLSI implementations (Mahowald, 1992). Thus, it is an event-driven asynchronous and digital multiplexing technique. The main functionality of AER circuits is to provide multiplexing / demultiplexing mechanisms for spikes that are generated by / on sent to a set of neurons. Fig. 1.6 schematically shows the transmission of information 1.2. Neuromorphic engineering 17 between two neuromorphic chips using the AER protocol. This protocol uses a shared multiplexed high-speed bus (AER bus) for transmitting the spikes that are fired by the neurons on a chip. Each neuron is identified with a unique address. Every time that a neuron spikes, and thanks to an arbiter circuit, the address of that neuron will be placed in the AER bus, generating an AER event. This way, each of the asynchronous spikes will be encoded and multiplexed by the AER circuit in the bus in the same order as they were generated (Serrano- Gotarredona et al., 2009; Berge and Hafliger, 2007). The timestamp in which the address of the neuron is generated corresponds to the timestamp in which it was fired plus a small delay caused by the codification process. These coding circuits use a specific arbitrating logic to handle the transmission of multiple spikes simultaneously from different neurons. Many different AER encoders can be found in the literature, based on the mechanism used to solve the conflict of multiple simultaneous spikes and how the addresses are encoded (Cerezuela- Escudero et al., 2013). Each of these options have their own advantages and disadvantages depending on the neuromorphic system used. A comprehensive study of these mechanism can be found in (Liu et al., 2015). FIGURE 1.6: Transmission of an event using AER representation. Image taken from (Lazzaro et al., 1993). The address of the neuron that produces the AER event is decoded on the receiver chip as soon as it arrives using an asynchronous decoder, sending the initial spike to the corresponding neuron. In this way, transmitter and receiver neurons are virtually connected through the AER bus using the AER protocol. If the delay between adjacent spikes in the input is high enough, the AER decoder will send the spikes to the corresponding neurons with a delay that corresponds to the time that the spike takes to reach the decoder plus a small delay caused by the decoding circuit. The AER protocol is a 4-phase asynchronous handshake protocol between the transmitter and the receiver that guarantees the synchronization between both chips, as shown in Fig. 1.6 bottom. The transmitter starts the communication 18 Chapter 1. Introduction process with a request. Then, the second chip, the receiver, answers to this request with an acknowledgement. To conclude the transmission, the transmitter removes the request and the receiver does the same with the acknowledgement, resetting the system to its initial condition. Both parts of the communication are not active until the transmitter starts a new request. A new transmission depends on the neurons of the transmitter chip that try to send and AER event. Thus, AER is a data-driven protocol. In this asynchronous protocol, the activity within the communication bus depends on the data transmissions, which is in contrast with classical discrete and periodic systems. With the increasing number of neuromorphic hardware including sensors and systems for deploying and running SNNs in real time, a proper set of software tools to process spiking information have become very useful for debugging tasks, dataset collection, and post-processing algorithms execution. One of the most popular software tool by the neuromorphic community is jAER (Delbruck, 2008). It is an open-source (under GNU Lesser General Public License v2.1) framework for PCs for visualization of real-time or recorded eventbased data1, and rapid development of real-time event-based algorithms and applications built in Java. jAER consists of an application called “jAERViewer” that allows to plug in any AER device with USB interface and perform different functionalities with it, e.g. view the events coming from the device in real time, log (record) them to a file (with .aedat extension), play a logged AER stream back and process the events using different filters. Fig. 1.7 shows a screenshot from AER, receiving spikes in real time from the live output of a Neuromorphic Auditory Sensor (NAS). There are many AER devices compatible with jAER, from dynamic vision and audio sensors to AER monitor/sequencer boards, along with a servo motor controller, among others. The events are produced by sensors asynchronously and timestamped (with 1 µs precision). Then, they are transmitted through the USB to the PC in packets, which contain variable numbers of events, and when they are received, jAER applies a set of filters chosen by the user. Meanwhile, the software can render the events that are output by the final filter and add visual annotations over the output. For offline representation and analysis of .aedat files, Neuromorphic Auditory VISualizer (NAVIS) tool was proposed in (Dominguez-Morales et al., 2017c) as a complement of jAER. NAVIS is open-source tool 2(under GNU General Public License) for post-processing the output spiking information from neuromorphic auditory sensors previously recorded with either Matlab or jAER. It allows you to load an events file and generate its full raster plot, so- 1With “event-based data” we mean address-events from systems using AER protocol which have been timestamped. 2NAVIS’s project and code can be found in https://github.com/jpdominguez/NAVIS-Tool 1.2. Neuromorphic engineering 19 FIGURE 1.7: Screenshot of NAVIS tool. called spikegram, histogram, sonogram, disparity, and mean activity plots. A spikegram example from NAVIS is shown in Fig. 1.8. FIGURE 1.8: Main window of NAVIS tool showing the cochleogram of the sentence "En un lugar de La Mancha". Blue dots represent the output of the left 64-channels, while orange dots represent the output for the right channels. Image taken from (Dominguez-Morales et al., 2017c). Furthermore, this tool includes some options for generating audio datasets, like an automatic splitter or mono-to-stereo/stereo-to-mono converters, among others. NAVIS has been succesfully used, for instance, in (Dominguez-Morales et al., 2018a), where authors first converted an audio dataset from heart recordings to spikes using NAS and jAER, and then NAVIS for generating the sonogram of each recorded .aedat file. 20 Chapter 1. Introduction Since NAVIS was implemented by using Windows Presentation Foundation (WPF), non-Windows users were not able to use it. Therefore, a cross-platform, open-source (under GNU General Public License) python version was developed by Dominguez-Morales et al. called pyNAVIS (Dominguez-Morales et al., 2021b) 3. As opposed to NAVIS, pyNAVIS does not have Graphical User Interface (GUI), so it is not as much interactive as NAS could be due to every action to be carried out must be already included in the script. However, the advantage of pyNAVIS is that it can be integrated into bigger scripts as a package, thus allowing the user to do multiple task but using the same Python script. 1.3 The sense of hearing In previous chapters, neuro-inspired systems were introduced. These systems mimic the way in which the senses and the brain process the information. Thus, it is intended to obtain benefits in the processing that are present in living organisms and that could be used for particular tasks like speech recognition and sound source localization. To be able to develop this kind of systems, it is necessary to have basic biological knowledge of the processing to be emulated. Therefore, in this chapter, the main characteristics of the sense of hearing are presented, as well as a state-of-the-art analysis of different bioinspired neuromorphic auditory systems that have already been published. 1.3.1 Auditory system in biology In this chapter, the anatomy and the physiology of the auditory system are described, emphasizing the parts and structures that are more relevant for audio processing and sound source localization. The concept of perception consists in the detection of a stimulus by one or more sensory receptors. In addition, interpretation, elaboration, and attentional selection are also components of perception. The sensory receptors of the human body are continuously sensing things that we are not even aware of (e.g., blood pressure, blood temperature, carbon dioxide and oxygen concentration, etc.) due to the fact that the signals that are sensed are not directly sent to the ensemble of network interactions that give rise to consciousness. In other circumstances, perception leads to sensation (or conscious perception). In these cases, the sensory receptors transmit impulses to the brain cortex through nerves and pathways. The brain cortex is able to integrate these signals into a sensation in the order of tens or hundreds milliseconds (Hull, 2011). Hearing is one of the senses that are classified as special senses, which have specialized organs devoted to them (vision has the eyes, hearing has the ears, smell has the nose, and taste has the tongue). Therefore, hearing could be 3pyNAVIS’s project and code can be found in https://github.com/jpdominguez/pyNAVIS 1.3. The sense of hearing 21 defined as the detection of sound waves and their integration in order to generate sensations (Hull, 2011). Every sensation, including the ones produced by hearing, are the result of the same sequence of events: first, a stimulus is produced. Then, the sensory receptor detects the stimulus and converts it into an electrical signal. Next, the signal is transmitted to the brain (through the auditory pathways in the case of hearing). Finally, the brain integrates the signal into conscious perception (sensation). The task of receiving the audio signal, processing it and transforming it into impulses are carried out inside the ear, while neural processing and sound recognition are performed in the brain. Thus, two main regions or sections can be distinguished in the auditory system: the peripheral auditory system, where sound waves are converted into nerve impulses, and the central auditory nervous system, which transforms these impulses into sensations. Different cognitive processes intervene in the central auditory nervous system, by which context and meaning are given to the sound, i.e., they allow the recognition of words, the classification of different musical instruments, the localization of the sound source, distinguishing between different speakers by their pitch and loudness, etc., among other complex tasks. 1.3.1.1 The ear The peripheral auditory system, also known as the ear, is responsible for the physiological processes of hearing. These processes allow the reception of sound, transforming it into electrical impulses that are then sent to the brain through the auditory nerves. The mechanical processing of the sound waves and their transformation into impulses are non-linear processes (Zwicker and Fastl, 2013), which hinders the characterization and modelling of the auditory perception. The peripheral auditory system is divided into three interconnected parts: the outer ear, the middle ear, and the inner ear. Next, the anatomy and functionalities of each of these three parts of the ear are described, along with the propagation and the processing of the sound across them. 1.3.1.1.1 Outer ear The outer ear is the external part of the ear and it consists of the auricle and the ear canal. The auricle is the visible part of the ear that resides outside the head (the word “ear” is also used to refer to this part alone) and it is also known as pinna. Its function is to gather the sound waves and guide them through the ear canal. When they hit the auricle, the sound waves are reflected and attenuated, which provides additional information to the brain for determining the direction of the sound that the auricle is receiving. The determination of the direction of the sound depends on the outer ear anatomy. Except for specific cases in 22 Chapter 1. Introduction FIGURE 1.9: Anatomy of the ear. Image taken from (Patton et al., 2012). which sounds come directly from towards, rearwards, above or below, the sound reaches the closest ear in a fraction of a second earlier and louder than in the furthest one. If the difference is higher than 10 µs (minimum human interaural time difference threshold), the brain is able to detect and interpret it. The ear canal is 2.5 cm long and it extends from the auricle to the eardrum, also called the tympanic membrane, in the middle ear. Its main function is to protect the middle ear and to maintain the middle ear at a stable temperature. It conducts the vibrations gathered in the auricle to the tympanic cavity, amplifying sounds with frequencies between 3 and 12 KHz. Each of these components are shown in Fig. 1.9. 1.3.1.1.2 Middle ear The middle ear is the part of the ear that is internal to the tympanic membrane, which separates this part from the outer ear (Fig. 1.9). The eardrum vibrates when it is hit by sound waves. The middle ear contains three small ossicles (malleus, incus, and stapes), which propagate the sound waves from the eardrum to the inner ear. The stapes are in contact with one of the fluids contained in the inner ear through the oval window. Thus, the ossicular chain acts as a mechanism to transform air vibrations into waves in the fluid and membranes of the inner ear. In order to achieve this, the air pressure inside the middle ear must be the same as the atmospheric pressure, which is possible thanks to the Eustachian tube (also known as auditory tube or pharyngotympanic tube), 1.3. The sense of hearing 23 connecting the middle ear with the nasopharynx and allowing the pressure to equalize between the middle ear and throat (Hull, 2011; Patton et al., 2012). The main functions of the middle ear are: • To increase the pressure received by the eardrum (impedance matching). This is essential, since the cochlea is full of liquid, instead of air, and the density and compressibility of the cochlear liquid (the perilymph) is almost four thousand times lower than that of the air. If we did not have a mechanism to increase the pressure inside the middle ear, only 0.1% of the tympanic pressure would reach the cochlea. This acts as a sound normalization step phase. To protect the inner ear structures from extremely loud sounds, the stapedius (the muscle that stabilizes the stapes), stiffens the ossicular chain by pulling the stapes away from the oval window of the cochlea, decreasing the transmission of vibrational energy to the cochlea for sounds below 1-2 kHz and above 85-90 dB. This mechanism is known as the stapedius reflex or acoustic reflex. • To reduce the transmission of low-frequency sounds, acting as a low-pass filter, with an attenuation of 15 dB per octave in the 1 kHz band. 1.3.1.1.3 Inner ear The inner ear is the innermost part of the peripheral auditory system. It consists of the vestibular system, which is dedicated to control balance, and the cochlea, dedicated to hearing. The cochlea, which is shown in Fig. 1.10 top left, is 32-35 mm long, 4 mm2to 1 mm2wide (from the base to the apex) spiral-shaped cavity filled with two different fluids. Fig. 1.10 shows a cross section of the cochlea in which three tubular ducts can be seen. The central one is the cochlear duct (also known as scala media) and contains endolymph. The second and third ducts, called vestibular duct and tympanic duct (or scala vestibuli and scala tympani, respectively), contain the same fluid, perilymph, due to the fact that they are interconnected through a small opening in the apex of the cochlea called helicotrema. The base of the stapes is in contact with the fluid of the vestibular duct through the oval window, while the tympanic duct terminates at the round window in the tympanic cavity, as shown in Fig. 1.9. The lining between the cochlear duct and the vestibular duct is known as the Reissner’s membrane, while the lining between the cochlear duct and the tympanic duct is called basilar membrane (Fig. 1.10) (Hull, 2011). The basilar membrane is a structure whose width and rigidity are not constant: it is broad and rigid near the oval window, and it gets thinner and more flexible near the apex of the cochlea. Rigidity decays almost exponentially with the distance from the oval window. This variation affects the propagation 30 Chapter 1. Introduction in the ear based on the area of the cortex that is stimulated. The auditory cortex also has neurons that respond best to the action potentials generated by high amplitude sounds, while others are more sensitive to low amplitude sounds. Part of the auditory cortex is responsible for giving sense to the information, discriminating between different sound patterns based on its variations in tone, frequency, intensity and direction. The auditory cortex also receives information from other regions of the brain such as the visual cortex and the somatosensory cortex, collaborating in the interpretation of the signals (Hull, 2011). The auditory system also transmits information from the auditory cortex to the cochlea through the efferent pathway (also known as the descending auditory pathway), giving feedback and modifying the analysis that is made in the cochlea in the form of frequency discrimination or non-linear amplification of quiet sounds. OHCs play an important role in this process (Cant and Benson, 2003). Humans are able to perceive sounds in a frequency region between 20 Hz and 20000 Hz, although we are more sensitive to the ones between 2000 Hz and 5000 Hz. This range differs between two different persons, shrinking during life, particularly the perception of high frequency sounds (Rodríguez Valiente et al., 2014). The frequency resolution of the ear depends on the intensity and frequency of the input sound in such a way that the lower the frequency of the input sound is, the lower the resolution will be. For tones around 200 Hz, we are able to discriminate sounds with an accuracy of 1 Hz. For higher frequencies, the frequency resolution increases. For tones around 10 kHz, the frequency resolution is 200 Hz. The reason for this is that the auditory system acts as a set of overlapped filters, in which the lower frequencies involve narrower filters and the higher frequencies involve wider filters (Janus, 2004). Regarding the duration of the sound, it only has an inferior limit. The shortest perceptible sound range between 10 and 40 ms (Bascuas, 1997). Twenty milliseconds is a widely used value when integrating information of sounds in auditory processing. 1.3.1.2.1 Cochlear nuclei The afferent nerve fibers from the cochlear cells are connected to the vestibulocochlear nerve, that terminate in the Cochlear Nucleus (CN). More specifically, those fibers project in a tonotopic pattern to the cochlear nuclei, distributing the projections between both the VCN and DCN. The former encodes the lower frequencies, whereas the latter encodes higher frequencies (see Fig. 1.15). In addition, each cochlear nerve fiber stimulate different areas within the CN, thus activating different types of neurons and therefore obtaining diverse response patterns. Consequently, it can be said that the auditory ascending 1.3. The sense of hearing 31 pathway is split into pathways which extract different cues from the input acoustic information. FIGURE 1.15: Cochlear nerve fibers terminate in the dorsal and ventral cochlear nuclei in a tonotopic organization. Image taken from (Kandel et al., 2000). For instance, the cells of the VCN are able to extract timing and spectral cues that are then sent to other auditory nuclei for further processing. Among those cells, bushy cells can be considered as the one of the most important cells in this nuclei. They project bilaterally to the Superior Olivary Complex (SOC), and the projections can be split in two parts: 1) through the MSO (larger spherical bushy cells) and 2) through the LSO and MNTB (smaller spherical bushy cells). In the first case, these cells are sensitive to low frequencies, and projects in a bilateral way to the MSO. The distribution of cells in the MSO forms a circtuit that is able to detect interaural time delays, so-called ITD, thus permitting the localization of sounds in the horizontal plane (low-frequency sounds). In the second case, those cells are sensitive to high frequencies, and project the LSO ipsilaterally. In addition, globular bushy cells excite neurons in the contralateral MNTB that in turn inhibit principal cells of the LSO. The pathways through the LSO are involved in the detection of interaural intensity differences, so-called ILD, and contribute to the localization of high-frequency sounds in the horizontal plane (Kandel et al., 2000). Stellate cells excite neurons in the ipsilateral dorsal cochlear nucleus, probably in the ipsilateral LSO, in the periolivary nuclei, and in the contralateral ventral nucleus of the lateral lemniscus through collaterals of axons that project to the contralateral Inferior Colliculus (IC). The tonotopic array of stellate cells encodes the spectra of sounds. Octopus cells excite targets in the contralateral periolivary region and the ventral nucleus of the lateral lemniscus. These neurons detect onset transients and periodicity in sounds and may be involved in the recognition of sound patterns. 32 Chapter 1. Introduction Octopus and bushy cells in the VCN are able to respond with exceptionally rapid and precisely timed synaptic potentials. These neurons have a low input resistance and rapid responsiveness and prevents repetitive firing. Neurons in the VCN are able to encode different features of sounds because of differences in the pattern of input and their biophysical properties. Octopus cells detect synchronous firing in cochlear nerve fibers with exceptional temporal precision. Individual octopus cells detect coincident firing in the relatively large number of cochlear nerve fibers (more than 60) that contact them. Coincident firing in large numbers of cochlear nerve fibers is produced by periodic sounds such as vowels and musical sounds and by the onset of broadband sounds found in consonants or clicks. Compared to octopus cells, bushy cells convey a more sharply tuned but less temporally precise version of the firing patterns of cochlear nerve fibers. The roughly 10 cochlear nerve fibers that terminate on each bushy cell deliver relatively large synaptic currents that require summation of only a few inputs to trigger an action potential. Two properties of bushy cells enable these neurons to encode with precision the detailed temporal structure of sounds. First, the cells’ low input resistance shortens the voltage changes produced by the incoming synaptic currents; their need to summate several inputs removes variability in the timing of firing in cochlear nerve fibers by averaging. Second, the temporal fine structure of sounds that bushy cells encode provides information about the relative time of arrival of inputs to the two ears and is used at the next synaptic stage to form a map of the Interaural Time Difference (ITD) that underlie the ability to localize sound sources in the horizontal plane. The detection of musical pitch also requires the encoding of the temporal fine structure of sounds, but whether that information is carried through octopus or bushy cells or through a combination of pathways is not known (Kandel et al., 2000). According to the studies, the main source of information about sound frequency that complements information derived from tonotopic maps is the timing of neural firing. Recordings made from neurons in the auditory nerve show phase locking, the consistent firing of a cell at the same phase of a sound wave (see Fig. 1.16). Considering a sound wave as a sinusoidal variation in air pressure, a phase-locked neuron would fire action potentials at either the peaks, the troughs, or some other constant location on the wave. At low frequencies, some neurons fire action potentials every time the sound has a particular phase (see Fig. 1.16a). This makes it easy to determine the frequency of the sound; it is the same as the frequency of the neuron’s action potentials (Bear et al., 2020). Phase locking can still occur even if an action potential has not fired on every cycle (see Fig.1.16b). For instance, a neuron may respond to a 1000 Hz sound with an action potential on only perhaps 25% of the cycles of the input, but those action potentials will always occur at the same phase of the sound. If you have 1.3. The sense of hearing 33 FIGURE 1.16: Phase locking in the response of auditory nerve fibers. Sound at a low frequency can elicit a phase-locked response, either (a) on every cycle of the stimulus or (b) on some fraction of the cycles. (c) At high frequencies, the response does not have a fixed-phase relationship to the stimulus. Image taken from (Bear et al., 2020). a group of such neurons, each responding to different cycles of the input signal, it is possible to have a response to every cycle (by some member of the group) and thus a measure of sound frequency. It is likely that intermediate sound frequencies are represented by the combined activity of a number of neurons, each of which fires in a phase-locked manner; this is called the volley principle. Phase locking occurs with sound waves up to about 4 kHz. Above this point, the action potentials fired by a neuron are at random phases of the sound wave (see Fig. 1.16c) because the intrinsic variability in the timing of the action potential becomes comparable to the time interval between successive cycles of the sound. In other words, the sound waves cycle too fast for the action potentials of single neurons to accurately represent their timing. Above 4 kHz, frequencies are represented by tonotopy alone. 1.3.1.2.2 Superior Olive 34 Chapter 1. Introduction It is important to recall that neurons in the cochlear nuclei only receive afferents from the ipsilateral auditory nerve. Thus, all of these cells are monaural neurons, meaning that they only respond to the sound presented to one ear. However, at all later stages of processing in the auditory system, there are binaural neurons whose responses are influenced by sound at both ears. The response properties of binaural neurons imply that they play an important role in sound localization in the horizontal plane. FIGURE 1.17: Phase-locked discharges of an auditory nerve fiber (A) and a spherical bushy cell of the AVCN (B). Image taken from (Schnupp et al., 2011). The first structure where binaural neurons are present is the Superior Olive, also known as Superior Olivary Complex (SOC). Neurons in the SOC receive input from CN on both sides of the brain stem (see Fig.1.14). Cells in the cochlear nuclei that project to the superior olive typically have responses phase locked to lower-frequency sound inputs. Fig. 1.17 shows an example of non phase-locked response from the auditory nerve fiber (A) versus phase-locked responses from spherical bushy cells (B). In many vertebrates, including mammals and birds, neurons in the SOC compare the activity of cells in the bilateral CN to locate sound sources. The SOC is a collection of brainstem nuclei that takes part in multiple aspects of hearing 1.3. The sense of hearing 35 and is an important component of the ascending and descending auditory pathways of the auditory system. Separate circuits detect interaural time and intensity differences. The SOC is divided into three primary nuclei: the MSO, LSO, and the MNTB, and several smaller periolivary nuclei. Fig. 1.18 shows those nuclei in the SOC as well as the cochlea and the cochlear nuclei. The color code indicates the frequency sensitiveness of each nuclei and its distribution, where HF means "high frequency" and LF means "low frequency". FIGURE 1.18: Schematic drawing of primary auditory sound localization circuits in the mammalian brainstem. Image taken from (Kandler et al., 2009). The MSO is thought to help locate the azimuth of a sound, i.e., the angle to the left or right where the sound source is located. Sound elevation cues are not processed in the olivary complex since the elevation is extracted by analyzing the spectral features of the input sound. Differences in arrival times at the ears are not represented at the cochlea. Instead, a map of interaural phase is created in the MSO by comparing the timing of firing in responses to sounds from the two ears. Sounds arrive at the near ear before they arrive at the far ear, with ITDs being directly related to the location of sound sources in the horizontal plane. The neurons in the MSO are sensitive to their own preferred ITDs. Although individual neurons may fail to fire at some cycles, the population of neurons represents the fine structure of sound waves by firing with every cycle. In so doing, these neurons carry information about the timing of inputs with every cycle of the sound. Sounds arriving from the side evoke phase-locked firing that is consistently earlier at the near ear than at the far ear, resulting in consistent ITDs. In 1948, Lloyd Jeffress suggested that an array of detectors of coincident inputs from the two ears, transmitted through delay lines comprising axons with systematically differing lengths, could form a map of ITDs and thus a map of the location of sound sources. Fig. 1.19 shows a schematic of the model proposed by Jeffress in (Jeffress, 1948). 36 Chapter 1. Introduction FIGURE 1.19: The tapped delay lines, coincidence detectors and coincidence counters (rate integrators) of the Jeffress model. Image taken from (Cariani, 2011). Humans are capable of using interaural time-of-arrival differences (ITDs) of as small as 10-20 microseconds to distinguish directional differences of sound sources in the horizontal plane as small as 1-2 degrees (azimuth). Typically, ITDs range from zero for sounds coming from directly in front to about 700 microseconds for sounds coming directly from either side. Traveling around the head takes about 700 µs, and the medial superior olive is able to distinguish time differences much smaller than this. In fact, it is observed that people can detect interaural differences down to 10 µs. However, this mechanism does not work with continuous tones at high frequencies. If a sound coming from the right has a frequency of 20000 Hz, it means that one cycle of the sound covers 1.7 cm. After a peak reaches the right ear, it would take less than 0.6 µs before a peak arrives at the left ear, thus receiving many peaks of such a high-frequency wave in that time period. Therefore, no longer is there a simple relationship between the direction the sound comes from and the arrival times of the peaks at the two ears. Interaural arrival time is not that useful for locating continuous sounds with frequencies so high that one cycle of the sound wave is smaller than the distance between your ears (i.e., greater than about 2000 Hz) (Bear et al., 2020). Fortunately, the brain has another process for sound localization at high 1.3. The sense of hearing 37 frequencies. An Interaural Level Difference (ILD) exists between the two ears because your head effectively casts a sound shadow (Bear et al., 2020). There is a direct relationship between the direction the sound comes from and the extent to which your head shadows the sound to one ear. If sound comes directly from the right, the left ear will hear a significantly lower intensity. With sound coming from straight ahead, the same intensity reaches the two ears, and with sound coming from intermediate directions, there are intermediate intensity differences. Neurons sensitive to differences in intensity can use this information to locate the sound. Intensity information cannot be used to locate sounds at lower frequencies because sound waves at these frequencies diffract around the head, and the intensities at the two ears are roughly equivalent. Therefore, there is no sound shadow at low frequencies. Although the LSO does not form a map of the location of sounds in the horizontal plane, it performs the first of several integrative steps that use ILDs to localize sounds. Neurons in this nucleus balance excitatory input from small spherical bushy cells and stellate cells in the ipsilateral VCN with inhibitory input from a disynaptic pathway that includes globular bushy cells in the contralateral VCN and principal neurons of the ipsilateral MNTB. Sounds that arise ipsilaterally generate relatively strong excitation and relatively weak inhibition whereas those that arise contralaterally generate stronger inhibition than excitation. Differences in the location of the sound sources that generate differences in the balance between excitation and inhibition are reflected in the firing rates of neurons in the LSO. Neurons in the LSO are thus activated more strongly by sounds from the ipsilateral than from the contralateral hemifield (Kandel et al., 2000). Summarizing the two processes for localizing sound in the horizontal plane: with sounds in the range of 20–2000 Hz, the process involves interaural time delay. From 2000–20,000 Hz, interaural intensity difference is used. Together, these two processes constitute the duplex theory of sound localization. 1.3.1.2.3 Inferior colliculus The Inferior Colliculus (IC) occupies a central position in the auditory pathway of all vertebrate animals because all auditory pathways ascending through the brain stem converge there 1.14. Cells in the VCN send out axons that project to the SOC on both sides of the brain stem. Axons of the olivary neurons ascend in the lateral lemniscus (a lemniscus is a collection of axons) and innervate the IC of the midbrain. Many efferents of the dorsal cochlear nucleus follow a route similar to the pathway from the VCN, but the dorsal path bypasses the superior olive. Although there are other routes from the cochlear nucleus to the IC, with additional intermediate relays, all ascending auditory pathways converge onto the IC. The neurons in the IC send out axons to the 38 Chapter 1. Introduction Medial Geniculate Nucleus (MGN) of the thalamus, which in turn projects to auditory cortex. Many neurons in the central nucleus carry information about the location of sound sources. The majority of these cells are sensitive to interaural time and intensity differences, which are known to be essential cues for localizing sounds in the azimuth. Neurons are also sensitive to spectral cues that localize sounds in the vertical plane. To localize sounds accurately, animals must ignore the re1ections of sounds from surrounding surfaces that arrive after the initial direct wave front. FIGURE 1.20: Sensory and cortical information flows in the SC. The superficial layer of the SC is directly connected to the optic tract, while the deep layers receive auditory (from the inferior colliculus) and somatosensory input. Cortical feedback moderates the processing in the deep layer and influences the motor outputs. Image taken from (Pavlou and Casey, 2010). It is also important to mention that the IC sends axons not only to the MGN but also to the Superior Colliculus (SC), where the integration of auditory and visual information occurs, and to the cerebellum, as shown in Fig. 1.20. In addition, there is extensive feedback in the auditory pathways. For instance, brain stem neurons send axons that contact outer hair cells, and auditory cortex sends axons to the acMGN and IC. The SC is critical for reflexive orienting movements of the head and eyes to acoustic and visual cues in space. By the time they reach the SC, binaural sound cues and the monaural spectral cues that underlie mammalian sound localization merge to create a spatial map of sound, an auditory map, in which neurons are 1.3. The sense of hearing 39 unambiguously tuned to specific sound directions. This convergence is critical, for the binaural level and timing differences alone cannot unambiguously code for a single position in space. The spectral cues that provide information about vertical location are essential. Different locations in the vertical plane can give rise to identical interaural time or intensity differences. Such a spatial map is formed both in birds and in some mammals (Kandel et al., 2000). Within the SC the auditory map is congruent with maps of visual space and the body surface. Unlike the visual and somatosensory maps, the auditory map is computed from a combination of cues that identify the specific position of a sound source in space, and is not based on the peripheral receptor surface. Auditory, visual, and somatosensory neurons in the SC all converge on output pathways in the same structure that controls orienting movements of the eyes, head, and external ears. The motor circuits of the SC are mapped with respect to motor targets in space, and are aligned with the sensory maps. Such sensory-motor correspondence facilitates the sensory guiding of movements (Naveros et al., 2019). 1.3.2 Implementations of the auditory system on circuits Artificial cochleae model the basilar membrane using a set of filters or resonators with cutoff/center frequencies (depending on whether low-pass filters or bandpass filters are implemented), which mimic the frequency distribution along the basilar membrane. Depending on the position of the basilar membrane, its flexibility and width changes. These changes are implemented by setting different parameters to the filters. As has been presented in previous sections, the biological cochlea is a very complex part of the inner ear, which means that modelling and implementing its functionality is not an easy task. This fact has resulted on artificial cochleae implementations that only model part of the characteristics of the biological cochlea. These characteristics are chosen depending on the application that researchers want to integrate in the system. Therefore, the majority of artificial cochleae do not achieve comparable results to the ones obtained by a biological cochlea. Since the first design of an artificial cochlea presented by Richard Lyon and Carver Mead in 1988 (Lyon and Mead, 1988), activities referred to the implementation of different mathematical models of the cochlea using analog VLSI have increased. Digital implementations using reconfigurable logic devices have also increased in popularity, although not as much as analog approaches. Even three decades after the first silicon cochlea, artificial cochleae are still far from being comparable to the biological cochlea, especially in terms of power consumption, frequency range, dynamic range of the input, or noise immunity. Taking into account that these models aim to emulate in the same way as a system 46 Chapter 1. Introduction El sistema auditivo: modelos e implementaciones 62 comunicación, y en la actualidad, este protocolo está altamente extendido en los desarrollos neuromórficos. En 1996, se realiza otra implementación de la cóclea artificial propuesta por Watts, expuesta en el trabajo (van Schaik et al. 1996). En la Figura 3.18 se muestra la distribución de las frecuencias de corte de esta implementación respecto a la distribución de las frecuencias de corte de la implementación de Watts. Se observa como en la implementación de van Schaik et al. existe uniformidad en la distribución exponencial de las frecuencias. Gracias a esta mejora, esta cóclea artificial permite procesamiento de sonidos biaurales, tal y como se muestra en los trabajos (Chan et al. 2007) y (Yu et al. 2009), en los que se usan dos cócleas artificiales de van Schaik, ampliadas con módulos más avanzados para gestionar los spikes de salida mediante el protocolo AER, para hacer experimentos de localización. La cóclea artificial expuesta en (Chan et al. 2007), también se ha usado en un sistema de clasificación entre dos tipos de sonidos: una palmada o bombo (Jackel et al. 2010). Esta cóclea ha sido ampliada en número de canales, tasa de eventos de salida y en el rango de frecuencias en el trabajo (Liu et al. 2010). Figura 3.18. Distribución de las frecuencias de corte (Hz) para la cóclea artificial de Watts et. al. (izquierda) y de van Schaik (derecha) (van Schaik et al. 1996). El principal inconveniente del diseño en cascada es su poca tolerancia a fallos ya que si un elemento falla, este error se propagará al resto de elementos posteriores. También, hay que destacar que cada segmento va a añadir un cierto retraso a la señal de entrada, que será inversamente proporcional a la frecuencia FIGURE 1.25: Cutoff frequencies (Hz) distribution in Watts’ artificial cochlea (left) and van Schaik’s (right). Image taken from (Van Schaik et al., 1996) reduces the dynamic range of the system. Some of these issues can be solved with a parallel topology or a 2-D topology. 120 Modelos e implementaciones del Sistema Auditivo de cócleas analógicas porque cada filtro actúa de modo independiente y para crear el mismo efecto de ‘pendiente pronunciada’ en las altas frecuencias, se necesitaría filtros de un orden mayor, lo cual implica un aumento considerable en el área y consumo del sistema. En la siguiente figura se compara la salida de un único filtro de segundo orden tanto en el modelo 1-D paralelo, Figura 52.a, como en el modelo 1-D en cascada, Figura 52.b. Figura 52. Salida de un filtro de segundo orden, en el modelo de cóclea paralelo (a) y en el modelo de cóclea en cascada (b). La cóclea bidimensional (2-D) modela tanto la propagación de la onda a lo largo de la membrana basilar como el movimiento del líquido interior de la cóclea y de la membrana basilar. Tiene en cuenta, por tanto, el desplazamiento longitudinal y vertical (ejes x e y). Los filtros vecinos se acoplan a través de un sistema de resistencias que modelan el líquido del interior de la cóclea. Este modelo combina las ventajas de las dos estructuras 1-D anteriores: el acople de los filtros en paralelo permite generar una pendiente pronunciada en las altas frecuencias a pesar de seguir siendo filtros de segundo orden; además se mejora la tolerancia a fallos y se evita la acumulación de retrasos propio de la estructura en cascada. Las implementaciones de este modelo son las más recientes: en 1992, Watts presenta una implementación con 50 etapas que mejora el rango dinámico, estabilidad y errores derivados de los FIGURE 1.26: Frequency response of a second order filter in a parallel topology (a) and in a cascade topology (b). Image taken from (Domínguez Morales, 2018) Parallel topologies are usually chosen for their ease of implementation. However, even though the issues of cascade topologies are not present in these models, parallel topologies are not the best option when implementing an analog cochlea because each filter acts independently from the rest and higher order filters are needed to produce the same “steep slope” effect in the high frequencies as in cascade topologies, which increments the power consumption of the system, requiring a larger silicon area. Fig. 1.26 presents a comparison between the outputs of a single second-order filter in a 1-D parallel topology (Fig. 1.26 (a)), and in a 1-D cascade topology (Fig. 1.26 (b)). The implementations presented in (Liu et al., 1991) and (Liu et al., 1992) are some examples of parallel topologies. Following the same architecture that was implemented in these works, in 1998 a new artificial cochlea was developed for pattern extraction in speech recognition tasks (Kumar et al., 1998). In 2016, 1.3. The sense of hearing 47 Minhao Yang, Chen-Han Chien, Tobias Delbruck and Shih-Chii Liu presented a 64-channel binaural cochlea with parallel asynchronous event output (Yang et al., 2016). In this model, each binaural channel performs feature extraction by analog bandpass filtering and, after that, filtered signals are encoded into events via Asynchronous Delta Modulation (ADM). Bidimensional (2-D) cochleae model both the propagation of the wave across the basilar membrane and the fluid motion within the cochlea and the basilar membrane. The structure of the filters in this type of artificial cochlea are based on Lyon and Katsiamis’ model, which is detailed in section 1.3.2.1.2. Neighbor filters are connected through a set of resistors that model the inner fluid of the cochlea. This architecture combines the benefits of the previous 1-D architectures presented before: the coupling of the filters in parallel allows to generate a steep slope in high frequencies despite being second-order filters; the fault tolerance is improved and the delay accumulation is avoided. In 1992, Watts presented a 50 stages (channels) implementation that improved the dynamic range, stability and errors caused by the transistors (Watts et al., 1992), achieving a uniform frequency response and good values in the quality factors of the filters (Q). (Fragnière, 1998) describes a 2-D cochlear model in which the pressure and voltage are mathematically analogous to the acceleration of the basilar membrane and the action potentials. Some authors have published recent implementations of this model (Van Schaik and Fragnière, 2001; Shiraishi, 2003; Hamilton et al., 2008; Wen and Boahen, 2009). The artificial cochleae implemented in the last two works mentioned before (Hamilton et al., 2008; Wen and Boahen, 2009), are active. An artificial cochlea is active when filters change dynamically depending on the input signal. Generally, the gain is increased for low frequencies and decreased for high frequencies. This behavior aims to model the functionality of the OHCs. These implementations use AGC to change the gain of the cochlea depending on the changes produced in the input signal. However, in these cases, the AGC also controls the quality factor (Q) of each section of the cochlea. This way, the gain and the bandwidth of each filter are changed dynamically based on the changes in the characteristics of the input sound. 1.3.2.1.4 Digital cochleae The need for more efficient systems with lesser power consumption and lesser cost led to the research and development of new digital design techniques and architectures. FPGAs are a very good option for the development of neuromorphic systems. They are flexible, robust to temperature changes, they have a better dynamic range, a better Signal-to-Noise Ratio (SNR)5, a simpler 5SNR, or Signal-to-Noise Ratio, is defined as the ratio of signal power to noise power, often expressed in decibels (dB). 48 Chapter 1. Introduction computer interface, the board can be used for different applications and their development time is much shorter compared to analog VLSI systems. However, they move away from the biological models. The first digital cochlea was implemented in 1992 using an ASIC (Summerfield and Lyon, 1992), Fig. 1.27. It contains 71 sections of cascade filters based on Lyon’s cochlear model. The output of each filter is connected to a HWR that mimics the functionality of OHCs along with an AGC. Therefore, even though the model in which it is based is not an active cochlea (it does not have automatic gain control), this digital implementation includes it. El sistema auditivo: modelos e implementaciones 65 3.4.4. Cócleas digitales La primera cóclea digital fue implementada en 1992 usando Circuito Integrado para Aplicaciones Específicas (ASIC) (Summerfield & Lyon 1992). Contiene 71 secciones de filtros en cascada siguiendo el modelo coclear de Lyon. La salida de cada filtro está conectada a un rectificador half-wave (HWR), que junto al bloque de control automático de ganancia (AGC) simulan la función de los OHCs. Por lo tanto, esta implementación incluye el control activo de la ganancia aunque el modelo en que se basa no lo contempla. Figura 3.20.Diagrama de bloques de la cóclea digital propuesta por Summerfield et.al. Imagen tomada de (Summerfield & Lyon 1992) La cóclea digital del trabajo (Jones et al. 2000) implementa el banco de filtros paso de banda de segundo orden en una FPGA. El banco de filtros sigue la estructura paralela y fue específicamente construido para extraer el tono de sonidos complejos. Aunque la cóclea digital tiene una arquitectura sencilla, implementa en detalle el modelo de los IHCs y las vías auditivas. En el trabajo (Leong et al. 2003) se presenta una implementación de 88 secciones en cascada mediante filtros IIR de segundo orden (Infinite Impulse Response). Este tipo de filtros son frecuentemente usados porque consiguen bandas más altas y estrechas con menor número de operaciones aritméticas. En la Figura FIGURE 1.27: Block diagram of the digital cochlea proposed by Summerfield et al. Image taken from (Summerfield and Lyon, 1992) The digital cochlea of (Jones et al., 2000) implemented a second-order bandpass filter bank with 30 filters in an FPGA. This filter bank follows a parallel architecture and it was specifically built for extracting the tone out of complex sounds. Although this digital cochlea has a simple architecture, it implements the model of IHC and auditory pathways in detail. In (Leong et al., 2003), an 88-sections cascade was implemented by using second order Infinite Impulse Response (IIR) filters. This kind of filters are able to obtain higher and narrower bands with less arithmetic operations. Fig. 1.28 shows the frequency response of some of the sections of the cochlea presented in that work. Parallel architectures achieve a higher speed in the output, but they have a limitation in terms of the cochlea size because the resources needed increase linearly with the number of stages in order to produce the steep slope behavior in the filters. However, in cascade architectures, the design limits are determined by the speed in the output, due to the fact that it decreases linearly with the number of stages. 1.3. The sense of hearing 49 634 EURASIP Journal on Applied Signal Processing 0102030405060708090 −0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5 Time Amplitude (a) Impulse response (software). 0102030405060708090 −0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5 Time Amplitude (b) Impulse response (hardware). 102103104 −60 −50 −40 −30 −20 −10 0 10 20 Frequency (Hz) Gain (dB) 102103104 −60 −50 −40 −30 −20 −10 0 10 20 Frequency (Hz) Gain (dB) 32 30 28 26 24 22 20 18 16 14 12 10 10 12 14 16 18 20 22 24 −100 −80 −60 −40 −20 0 20 40 Quantization error (dB) Width of LUT Wordlength Figure 7: Mesh plot showing the quantization errors of implementations with varying wordlengths and DA ROM widths. 5. RESULTS The cochlea implementation was tested on an Annapolis “Wildstar” Reconfigurable Computing Engine [23]wh ich is a PCI-based reconfigurable computing platform containing three Xilinx Virtex XCV1000-BG560-6 FPGAs. The cochlea implementations were verified by comparing Synopsys VHDL Simulator simulations with the results produced by a floating-point software model. Synthesis and implementation were performed using Synopsys FPGA Express 3.4 and Xilinx Foundation 3.2i, respectively. 5.1. Trade-offs among wordlength, width of DA ROM, and precision The coefficients for the biquadratic filters in our implementation of Lyon and Mead’s cochlea model were obtained using Slaney’s Auditory Toolbox [24]. This Matlab toolbox has several different cochlea models, test inputs, and visualization tools. The same toolbox was used to verify our designs and produce cochleagram plots. The coefficients of these implementations were obtained from the Auditory Toolbox using the Matlab command DesignLyonFilters(16000, 8, 0.25), which specifies a 16 kHz sampling rate, Q=8, and a spacing which gives 88 biquadratic filters. A series of cochlea implementations, with wordlengths from 10 to 32 bits and DA ROM width from 10 to 24 bits, was generated in order to present the trade-offs among wordlengths, widths of DA ROMs, and precisions. In order to present the improvement in precision with increasing wordlengths and ROM width, the frequency responses of several different fixed-point implementations are plotted in Figure 5. Figure 6shows impulse and frequency responses obtained from a software floating-point implementation, a hardware 16-bit wordlength, and 16-bit ROM width implementation. It can be observed that the filter accuracy gradually improves with increasing wordlength or ROM width. When (b) Frequency response(hardware) (a) Frequency response (software) FIGURE 1.28: Frequency response of the digital cochlea implementation by Leong et al. Image taken from. Image taken from (Leong et al., 2003) Some recent implementations of digital cochleae use IIR filters, and both parallel (Dundur et al., 2008; Miró Amarante, 2013) and cascade (Gambin et al., 2010; Mugliette et al., 2011; Thakur et al., 2014) implementations can be found. The digital design presented in (Dundur et al., 2008) is the basis for the cochlear implant implementations in FPGAs. Other designs (Gambin et al., 2010; Mugliette et al., 2011; Thakur et al., 2014) use time multiplexing to implement the filter bank with a greater number of sections. For example, the digital cochlea presented in (Mugliette et al., 2011) has a cascade of 24 second-order IIR filters, and thanks to the time multiplexing, it only requires 20 multiplier units. Designs that are based on time multiplexing use less resources in exchange of using higher clock frequencies, increasing the power consumption of the system. Using FPGA implementations have some advantages over VLSI analog systems: a faster design time, more robustness to power supply changes, to temperatures and to transistor mismatch, a higher dynamic range, a higher SNR, better stability, FPGAs can be reused for different applications and they have a simpler interface to the computer. Two of the most recent digital cochlea implementations in FPGA are (Jimenez-Fernandez et al., 2017) and (Xu et al., 2018a). In (Xu et al., 2018a), a digital implementation of a 70-section, 44.1 kHz sampling rate Cascade of Asymmetric Resonators with Fast-Acting Compression (CAR-FAC) (Lyon, 2017) cochlear model is presented, where the Cascade of Asymmetric Resonators (CAR) part simulates the basilar membrane’s response to sound and the Fast- Acting Compression (FAC) part models the OHCs, the IHCs, and the medial olivocochlear efferent system functions. The FAC feeds back to the CAR by moving the poles and zeros of the CAR resonators automatically, making it an active cochlea model. 50 Chapter 1. Introduction The digital implementation proposed in (Jimenez-Fernandez et al., 2017), which is called Neuromorphic Auditory Sensor (NAS), is the one that has been used throughout this thesis. NAS (Jimenez-Fernandez et al., 2017) is an audio sensor for FPGAs inspired by Lyon’s model of the biological cochlea (Lyon and Mead, 1988). This sensor is able to process an excitatory audio signal using Spike Signal Processing (SSP) techniques (Jimenez-Fernandez et al., 2010), decomposing incoming audio in its frequency components, and providing this information as a stream of events using the AER (Boahen, 2000). As it is implemented on a reconfigurable platform, this sensor’s configuration parameters are flexible and can be adapted to any application. NAS decomposes two digitized signals (two in case of a binaural or stereo NAS, or one in case of a monaural or mono NAS) into a set of sections or bands (corresponding to particular frequencies), which are previously converted to a spike train. The decomposition of the signals in each corresponding frequency band is performed by a bank of Spike-based Low-Pass Filter (SLPF) filters (Jimenez-Fernandez et al., 2010; Domínguez-Morales et al., 2011) connected in a cascade topology. The output of this sensor is encoded using the AER protocol as many of the neuromorphic sensors that were presented in the previous section. For this decomposition of the input signals (left and right ear) the same processing is performed. The processing is modeled using a Spike-based Cascade Filter Bank (SCFB). Each SCFB has many stages (as many as sections in the cochlea) and consists of a SLPF and a Spike Hold&Fire (SHF) (Jimenez-Fernandez et al., 2010), which are able to extract two different spike trains: one that is passed as input to the next stage and another one that corresponds to the signal filtered by the current stage of the cascade. This kind of sensor mimics how the biological cochlea processes audio signals. NAS is able to decompose the input audio signal into different frequency bands (also called channels). Like in other neuromorphic cochleae implementations based on a cascade model (Liu et al., 2010b; Leong et al., 2003; Summerfield and Lyon, 1992; Lyon and Mead, 1988), this decomposition is carried out by a series of cascade-connected stages that subtract the information from consecutive spike-based low-pass filters’ output spikes in order to reject out- of-band frequencies, obtaining a response equivalent to that of a bandpass filter (Jimenez-Fernandez et al., 2017). Fig. 1.29 (a) shows the global architecture of a binaural NAS. In this architecture, the first element in the chain is an AC97 audio codec (CS5344 audio codec), which has two analog signals as inputs (left and right) that are digitized and multiplexed into a single output. This signal is divided in order to obtain sampled and digitized values that correspond to each of the input signals. After that, the information is converted from a digital domain to a spike domain by using a Reverse Bitwise Synthetic Spikes Generator (RB-SSG) (Jimenez-Fernandez et al., 2010), which provides a spike stream with a frequency 1.3. The sense of hearing 51 806 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 28, NO. 4, APRIL 2017 Fig. 1. (a) Global NAS architecture. (b) Filter banks with Cascade topology, CFB. (c) Single CFB stage containing an SLPF and an SH&F. small general purpose building blocks. Each of these blocks performs a specific primitive arithmetic operation on the spike streams, and can be combined with others to build large spike processing systems of the type already used in closed-loop spike-based PID controllers [32], and trajectory generators for object tracking [33]. A. Reverse Bitwise Synthetic Spike Generator Digital sound samples received from a commercial audio codec were immediately converted into a stream of spikes by a digital synthetic spike generator (SSG) capable of converting a discrete number (SSG input) into a fixed spike frequency rate (SSG output). These output spikes represented the audio information that would excite the CFB. This SSG is also the formed part of other elements in the CFB capable of processing spike-coded signals. This will be explained later. Although there are several ways to design a digital SSG [30], [34], the implementation in this paper used the reverse bitwise method for synthetic AER event generation [reverse bitwise SSG (RBSSG)] described in [30] and [35]. Fig. 2. RBSSG [10]. This architecture was selected mainly for its low resource needs (a digital counter and a comparator) and closer to uniform temporal spike distribution. An SSG would generally be capable of generating a synthetic spike stream with a frequency proportional to a constant (kBWSpikesGen)and an input value (x),asin RBSSG(x)SpikesRate =kBWSpikeGen ∗x.(1) Fig. 2 shows the RBSSG circuit. It uses a continuous digital counter [Fig. 2 (top)], the output of which is reversed FIGURE 1.29: (a) Global NAS architecture. (b) Filter banks with Cascade topology, CFB. (c) Single CFB stage containing an SLPF and an SH&F. Image taken from. Image taken from (Jimenez-Fernandez et al., 2017) that is proportional to the digital amplitude. At the output spike generator, ON and OFF spikes that encode the previously sampled digital value are obtained. The generated spike train is the input to the cascaded spike-based filter bank Cascade Filter Bank (CFB) (depending on the NAS architecture implemented on the FPGA, the number of filters will be different). Fig. 1.29 (b) shows how these filters are distributed, forming the bank, where the first filters of the cascade correspond to higher frequencies, and the last ones correspond to lower frequencies. As was previously mentioned, the filter bank is divided into different stages, where each stage consists of a spiking low-pass filter along with a SHF (see Fig. 1.29 (c)). The output of each of the stages is connected to an AER monitor (Cerezuela-Escudero et al., 2013) that encodes the activity of each stage, placing the address of the corresponding stage in the output asynchronous AER bus every time that an event is generated in it. 53 Chapter 2 Objectives “I have no special talent. I am only passionately curious.” – Albert Einstein In this chapter, the main objectives to be achieved in this thesis are listed. These objectives are divided in general objectives, presenting a global idea of the thesis; and specific objectives, where the corresponding tasks to achieve the general objectives are detailed. Then, the organization of this thesis is presented by giving a brief description of each chapter and appendix. 2.1 Objectives According to the proposed improvements and contributions mentioned in the motivation section 1.1, two types of objectives were proposed: general objectives, with the aim of analyzing and studying the viability of a fully digital, eventbased auditory system model and its implementation in both ASIC and FPGAs, for being integrated afterwards within robotic platforms; and more specific objectives, focused on solving particular problems related to the topic introduced previously and the design of the corresponding systems and models. General objectives: 1. Develop a set of tools to provide the community to have access to a neuromorphic auditory sensor through an open-source project. 2. Study how the human auditory system captures, analyzes and encodes binaural information into nerve impulses that are processed by the brain when carrying out sound source localization tasks. 3. Integrate the auditory sensor in both general and specific-purpose robots for performing audio processing, as well as sensory integration, in realtime. 54 Chapter 2. Objectives With these general objectives, it is intended to mimic the human nervous system to design neuromorphic systems, as well as to explore the benefits of the auditory processing combined with other sensory information, like vision, in robotics. To achieve these general objectives, the following set of specific objectives were proposed: Specific objectives: 1. Open-source Neuromorphic Auditory Sensor (NAS) for FPGAs and ASICs implementations: (a) Study of how NAS works, its parameters, and the behavior of its internal modules when an input stimulus is presented. (b) Development of a new tool for automatically generating HDL files to implement a NAS according to a list of configuration parameters specified by the user. (c) Study of the digital ASIC design workflow from RTL description to the signoff process. (d) Design of an ASIC containing a tiny NAS model as a proof-of- concept of the first implementation of a digital, fully event-based neuromorphic auditory sensor in a chip. 2. Neuromorphic models for binaural cues extraction and sound source localization: (a) Study the biological and physiological principles of the binaural hearing in humans from the ear to the auditory cortex, and the effect of the visual attention to the auditory perception. (b) Study and analysis of state-of-the-art Jeffress model implementations for the ITD extraction. (c) Design and development of a digital, event-based Jeffress model for FPGAs. (d) Design and development of a digital, event-based model of a simplified human Superior Olivary Complex (SOC) model for performing the binaural cues extraction. (e) Integration of the proposed SOC model with the NAS model in order to contribute to the implementation of a neuromorphic hearing sense. (f) Characterization and evaluation of the integration under several conditions for analyzing its scalability and adaptability. (g) Study of novel alternatives of binaural cues extraction by taking inspiration from the optical flow estimation process in insects. (h) Design and implementation of the TDE neuron model for FPGAs. 2.2. Thesis structure 55 (i) Test, analysis and evaluation of the use of the TDE model for the ITD extraction as an alternative to the Jeffress model for the sound source lateralization task. 3. Integration of neuromorphic auditory systems for real-time audio processing in robotics: (a) Study of the relation between the sound recognition task and Central Pattern Generators (CPGs) in biology and the state-of-the-art implementations. (b) Design of an audio-guided SCPG based on the central nervous system of insects on SpiNNaker. (c) Implementation of a neuromorphic robotic platform for performing real-time test about locomotion changes depending on the recognition of different auditory input stimuli. (d) Study of the audio-visual integration process in the brain. (e) Design, implementation and test of a neuromorphic robotic system to integrate event-based sensory information for attention-related tasks. (f) Integration of the proposed neuromorphic auditory sensor within the iCub robotic platform. (g) Evaluation of the integration, identification of improvements, and implementation of a demonstration of real-time audio processing in iCub. 2.2 Thesis structure The thesis is structured in three chapters, following the specific objectives previously detailed, as well as a conclusions and future works chapter. In addition, three appendices are included containing useful information for the reader, such as software manuals and design files. Each chapter and appendix are briefly described below: • Chapter 3: Open-source Neuromorphic Auditory Sensor (NAS) for FPGAs and ASICs implementations. In this chapter, OpenNAS tool is presented. This tool has been developed to provide an open-source alternative of an event-based auditory sensor to neuromorphic researchers. The tool architecture, workflow, and performance are described also in this chapter. In addition, the preliminary results of the first ASIC implementation of a NAS are showed. • Chapter 4: Neuromorphic models for binaural cues extraction and sound source localization. 62 Chapter 3. Open-source Neuromorphic Auditory Sensor OpenNAS was designed following a hierarchical class structure to represent NAS’ components, with its HDL description and parameters, allowing developers to increase OpenNAS components and functionalities easily. Fig. 3.4 shows a summary of the OpenNAS usage flow from the download of the repository until the FPGA programming. 3.3 NASIC As introduced in Section 3.2, FPGA-based solutions can be considered as one of the best options for testing and validating research developments and proof-of- concept designs. This approach is also efficient in cases that the reconfigurability of the design plays a key role, since this kind of platforms are capable of updating part of its deployed design in real time (Liu et al., 2009; Sabena et al., 2014; Montealegre et al., 2015). Nevertheless, ASIC solutions could provide advantages for embedded applications due to its low-power consumption, small size, and applicationspecific input/output interfaces. For instance, a NAS ASIC would be needed in the future for the first generation of neuromorphic cochlear implants (Lande et al., 2000). In addition, ASICs are desirable in robotic platforms, like iCub (Natale et al., 2017), where many different parts and components have to fit in a reduced space. For this purpose, this thesis presents an ASIC which implements a tiny model of the Neuromorphic Auditory Sensor (NAS) as a proof-of-concept, also known as NASIC. The NAS was generated using the OpenNAS tool, and to the best of our knowledge, it is the first time that a full event-based, digital Neuromorphic Auditory Sensor (NAS) is produced as an ASIC. The project’s repository is open-source, and it can be found on GitHub 3. 3.3.1 Motivation The NAS was already used within robotic applications, either integrated or not into the platform itself, as presented in (Schoepe et al., 2019; Gutierrez-Galan et al., 2019a; Cerezuela-Escudero et al., 2018). In those cases, neither the power consumption nor the high performance was a crucial aspect to take into account since the main goal was the applications themselves. Recent works in the field of e-health and biomedical engineering, where the NAS was the main component (Dominguez-Morales et al., 2017b; Dominguez- Morales et al., 2021a), motivated the decision of designing, fabricating, and testing a NAS ASIC. This way, it would be possible to think of commercial applications including neuromorphic chips for health-related tasks in hospitals. 3https://github.com/dgutierrezATC/NASIC 3.3. NASIC 63 In this sense, analog neuromorphic designs are somehow forced to be produced as ASICs to be tested and used in real-time, low-power applications. Although there is reconfigurable alternatives, as the Field-Programmable Analog Array (FPAA) (George et al., 2016) that offers the possibility to deploy custom analog designs in a reconfigurable floating-gates fabric, researchers often prefer to fabricate their own chips to really take advantage of the technology, such as the low-power consumption (Milde et al., 2018). For instance, there are analog cochleae ASICs, as presented in Section 1.3.2.1.3, which have been successfully used in many projects, demonstrating their low-power features and good performance. Therefore, we would like to compare the performance of our proposed digital, fully event-based cochlea model in an ASIC with the existing state-of- the-art silicon cochleae. Finally, one of the main goals is to study the ASIC design flow for adding new functionalities to OpenNAS in order to automatically generate common ASIC design files, such as the constraints file or testbenches. This way, the OpenNAS tool would be involved in future NASIC design and fabrication. 3.3.2 Base architecture The OpenNAS tool was used for generating the NAS’ HDL files. Since this was the first ASIC designed by the RTC Research Lab., a tiny version of a NAS was desired to minimize potential risks, such as timing problems, size restrictions, etc., and to reduce simulation and synthesis times. As it was mentioned before, the main goal is to check whether the performance of an ASIC-based NAS is similar or not when compared to the performance of an FPGA-based NAS. Table 3.2 summarizes the selected values for the base project, organized according to the OpenNAS parameters. An eight-frequency-channels, mono, cascade Neuromorphic Auditory Sensor (NAS) was generated, setting the working frequency range between 20 Hz and 22000 Hz. The main clock frequency was set to 48 MHz, that is the same clock frequency of the FPGA-based board ZTEX 2.13, which has been used to test the same files on FPGA prior to design the chip. As input interfaces, both I2S- based interface and Pulse-Density Modulation (PDM)-based microphones were added for testing the NAS’ behavior with different stimuli sources. As output interface, a distributed events monitor was included for providing an AER interface to send the NAS’ output events. However, this module was removed from the design due to timing problems to satisfy the timing constraints. Therefore, no output interface was included in this first version. Instead, two dedicated pins are used for each filter’s output, having sixteen pins in total for the output events (eight frequency channels, with positive and negative events for each channel). Thus, the absence of events monitor implies the use of an 64 Chapter 3. Open-source Neuromorphic Auditory Sensor NASIC Spikebased cascade bandpass filter bank I2S interface PDM interface Input interface selector Spike generator Spike generator From audio ADC From PDM mic. Selector Raw output spikes FIGURE 3.5: NASIC’s internal architecture block diagram. external FPGA-based board for implementing both the events monitor and the AER protocol. TABLE 3.2: NASIC features Parameter Value Notes NAS chip Other 180 nm TSMC NAS type Mono Num. of channels 8 NAS common settings Clock freq. (MHz) 48 Input interface Audio input I2S + PDM All the values were by default. NAS architecture Cascade Start frequency 20 Hz End frequency 22000 Hz Processing architecture SLPF output att. -15 dB Output interface Spikes output Raw No AER monitor was integrated. The NAS base project was deployed into an FPGA and characterized in order to set a ground truth for being used as reference when validating the ASIC. For this characterization, a set of six pure tones were generated and used as input of the NAS. The pure tone frequencies are identical to the ones used in (Dominguez- Morales et al., 2016). The duration of each pure tone was two seconds, and the computer’s volume was set to 60%. A high speed audio Analog-to-Digital Converter (ADC) with Integrated Interchip Sound (I2S) interface was used, since it allows to connect an 3.3. NASIC 65 audio cable directly from the computer to the NAS. Fig. 3.6 shows the histograms of the pure tones used.                  (A) 261 Hz.                  (B) 349 Hz.                  (C) 523 Hz.                   (D) 698 Hz.                  (E) 1046 Hz.                 (F) 1396 Hz. FIGURE 3.6: Histograms from pure tones using NAS on FPGA. Middle frequencies associated to each frequency channel were: channel 0, 22000 Hz; channel 1, 8090 Hz; channel 2, 2975 Hz; channel 3, 1094 Hz; channel 4, 402 Hz; channel 5, 148 Hz; channel 6, 54 Hz; and channel 7, 20 Hz. As it can be seen, the main activity peak for each case is shifted from low frequencies (higher addresses) to high frequencies (low addresses) according to 66 Chapter 3. Open-source Neuromorphic Auditory Sensor the input stimulus frequency. The shape of the histogram keeps almost identical in all cases. However, the average activity, expressed as the number of events per second, varies between cases. The case 3.6a had about 325000 events per second, while the case 3.6f had around 250000 events per second. This effect is produced due to, for low frequency sounds, the lasts bandpass filters of the cascade filter bank are the ones tuned to have a better response to those sounds, in the same way to the basilar membrane. Therefore, the sound will go through the filters in the filter bank, exciting all of them and, thus, producing more spikes. In the opposite way, high frequency sounds will stimulate the filters placed at the beginning of the filter bank, thus not stimulating the low-frequency tuned bandpass filters, which are placed at the end of the filter bank. Fig. 3.7 shows the sonogram of each pure tone obtained from the NAS model before designing the ASIC. The correlation between the histograms and the sonograms is perceptible. It can also be seen how the main frequency band, i.e., the frequency band with the higher activity, changes according to the input sound, as it was already observed in the histograms. 3.3.3 NASIC design process In this design, a 180 nm process technology from Taiwan Semiconductor Manufacturing Company (TSMC) was used. As Electronic Design Automation (EDA) software, Design Compiler from Synopsys was used for performing the synthesis, while Innovus from Cadence was used for carrying out the physical implementation. The simulations were run using ModelSim from Intel FPGA. First, the synthesis tool, Design Compiler in this case, was set up. The search paths for specifying the directories which contain the core and I/O libraries were set, among others. After configuring the synthesis tool, the next step was to load the design. This process is a two-stage process. The first one is the analysis stage, where the software analyzes the RTL HDL code and compiles it. If the design had any problem, an error would be raised. The second stage is the design elaboration, where the tool, after analyzing the code, creates the design and loads it in memory. In this process, much information and feedback is shown, including state machine extraction, register inference, and design hierarchy. Several changes were carried out due to some design errors found at the elaboration step. At this point, a netlist was generated but without any link to the process technology library. As a first check, this generated netlist was used as the main component in the simulation. First, the original HDL code generated by OpenNAS was simulated, and the resultant simulation file, which includes the NAS’ event address and its timestamp, was saved. Then, the elaborated netlist was also simulated. Finally, both simulation files were compared, perfectly matching in both addresses and timestamps values. 3.3. NASIC 67                             (A) 261 Hz.                             (B) 349 Hz.                            (C) 523 Hz.                             (D) 698 Hz.                             (E) 1046 Hz.                           (F) 1396 Hz. FIGURE 3.7: Sonograms from pure tones using NAS on FPGA. When the generic netlist was ready, the mapping (to cells) and optimisation procedures could be applied. Since the timing constraints were not specified at this point, no timing optimisations were performed. Instead, an area reduction optimisation was done, having a first approximation of 0.5 ×0.5mm2. 68 Chapter 3. Open-source Neuromorphic Auditory Sensor FIGURE 3.8: ASIC design workflow. Image taken from https://www. einfochips.com/ The next step was to constraint the design by modeling the clock and constraining both the inputs and outputs. First, the clock was created, being defined at 48 MHz. Just after the clock definition, a first timing test was performed, verifying that most of the paths were constrained by the clock. Then, the rest of the clock effects were modelled by defining the clock latency, the clock uncertainty, and the clock transition. Finally, the I/O timing constraints were set, and a timing verification was run. Several timing problems were identified at this point, which were located in two different parts of the design. On the one hand, there was inconsistency with the reset signal, which was used sometimes low active and sometimes high active. On the other hand, the AER events monitor, that was originally integrated on the design, presented timing problems since its FIFO memories were implemented as registers instead of memory cells, as it is synthesized in an FPGA. The reset issue was solved, but the events monitor problem was not solved and, therefore, it was removed. After constraining the design, and once the timing problems were solved, a timing optimisation was performed. A new simulation was carried out in order to check that the changes applied to the design did not affect its behavior, being the result of the simulation identical to the reference. The procedures and function calls were automated by using TCL scripts, 3.3. NASIC 69 allowing us to repeat the whole design flow every time a change was done. These TCL scripts are being integrated in OpenNAS with the aim to facilitate the NASIC design process in the future. For performing the floorplan step, we moved from Design Compiler (Synopsys) to Innovus (Cadence). Therefore, the design was exported from the first software tool and imported in the second software tool. The global configuration was created, as well as the physical constraints (core utilization, boundaries, power ring, etc). FIGURE 3.9: NASIC cells distribution after the place & route process. Input and output pads (known as IO pads) were also defined and placed at this stage. In turn, IO pads could be split depending on their use, i.e., if the pad was going to be used for the clock, power supply, or for other signals. Fig. 3.9 shows the distribution of the IO pads on the chip. After the physical constraints were set, the place & route process was carried out. The design’s cells were loaded, placed, and distributed over the core area available, and routed. Although the area estimation during the synthesis was 0.5 ×0.5mm2, a core size of 1 ×1mm2was set to facilitate the place & route 70 Chapter 3. Open-source Neuromorphic Auditory Sensor process. Finally, the cells were connected to either the IO pads or power supply rings, if needed, distributed over the core as shown in Fig. 3.9. For verification purposes, another simulation was performed after the place and route process. In this case, the cells’ library was used, containing the real timing information of the physical cells. Both the number of output spikes and their order were identical compared to the reference. However, small time differences (a few nanoseconds) were obtained. The last step was to fill the empty space inside the core with filler cells to avoid planarity problems and to ensure good continuity. After that, the final compilation was performed applying area, power, and timing optimizations. FIGURE 3.10: NASIC layout. Fig. 3.10 shows the final design layout, including the bonding pads. Thirty two pins were needed according to the number of IO pads used during the ASIC design process. However, a sixty four pins Ceramic Quad Flat Package (CQFP) package was used due to packaging limitations. Table 3.3 summarizes the NASIC’s pinout. As it can be observed, half of the pins were not connected internally to any circuit. For simplification purposes, the pin usage distribution was set in an alternate way, leaving one empty pin between each used pin. 3.3. NASIC 71 TABLE 3.3: NASIC chip pinout. Pin number Pin name Type Typical value Notes 0 VSS_CORE_2 Input GND Digital GND. 2 VDD_CORE_2 Input 1.8V Input current up to 1 A. 4 SPIKES_OUT_5 Output 3.3V 6 SPIKES_OUT_4 Output 3.3V 8 SPIKES_OUT_3 Output 3.3V 10 SPIKES_OUT_2 Output 3.3V 12 SPIKES_OUT_1 Output 3.3V 14 SPIKES_OUT_0 Output 3.3V 16 VSS_1 Input GND Digital GND. 18 VDD_1 Input 3.3V Input current up to 1 A. 20 PDM_I2S_SEL Input 3.3V 0 for I2S; 1 for PDM. 22 PDM_DATA_LEFT Input 3.3V 24 PDM_CLK_LEFT Output 3.3V 26 I2S_SD Input 3.3V 28 I2S_WS Input 3.3V 30 I2S_SCLK Input 3.3V To add a 2.7 kOhm pull-down resistor. 32 VSS_CORE_1 Input GND Digital GND. 34 VDD_CORE_1 Input 1.8V Input current up to 1 A. 36 CLK Input 3.3V The clock frequency is 48 MHz. 38 RST Input 3.3V Low active reset. 40 SPIKES_OUT_15 Output 3.3V 42 SPIKES_OUT_14 Output 3.3V 44 SPIKES_OUT_13 Output 3.3V 46 SPIKES_OUT_12 Output 3.3V 48 VSS_1 Input GND Digital GND. 50 VDD_1 Input 3.3V Input current up to 1 A. 52 SPIKES_OUT_11 Output 3.3V 54 SPIKES_OUT_10 Output 3.3V 56 SPIKES_OUT_9 Output 3.3V 58 SPIKES_OUT_8 Output 3.3V 60 SPIKES_OUT_7 Output 3.3V 62 SPIKES_OUT_6 Output 3.3V 78 Chapter 3. Open-source Neuromorphic Auditory Sensor Finally, an ASIC design would be a good option for commercial purposes and applications, since mass fabrication would decrease production costs. In this case, many factors should be taken into account, like the supply and demand ratio, target consumer, sales volume, etc. Nevertheless, the fact of creating an ASIC, although it was not perfectly working or the sales volume did not reach the expectations, could open new paths for future neuromorphic solutions to audiorelated problems in humans. 79 Chapter 4 Event-based models for the sound source localization task “The art of conversation is the art of hearing as well as of being heard” – William Hazlitt 4.1 Introduction Besides the cochlea, the auditory ascending pathway contains other nuclei that extract important features from the input sound. Among them, nuclei located at the Superior Olivary Complex (SOC) are capable to measure both the Interaural Time Difference (ITD) and the Interaural Level Difference (ILD). Those measurements, also known as binaural cues, play a crucial role in the sound source localization task since they provide useful information about the auditory scene (Kandel et al., 2000). In addition, sound source localization can be considered one of the most remarkable survival skills, since it allows to identify incoming risks in short time periods, i.e., hundreds of microseconds. In fact, auditory attention and audiovisual stimuli integration also use this capability, thus, being considered as an important part of the brain. According to biology, the SOC in humans, shown in Fig. 4.1 is composed by three main nuclei, which are the Medial Superior Olive (MSO), the Medial Nucleus of the Trapezoid Body (MNTB), and the Lateral Superior Olive (LSO), among others. As it was mentioned in Chapter 1, the MSO extracts ITDs, while the LSO extracts ILDs. Many works can be found in the literature, either in-deep studying how those nuclei work or implementing their functionality for specific applications (Bhadkamkar and Fowler, 1993; Lazzaro and Mead, 1989b; Liu et al., 2010a; Dávila-Chacón et al., 2018). The model proposed by Jeffress in (Jeffress, 1948) was a first approach for modelling the MSO. However, the model has been improved over the years, 80 Chapter 4. Event-based models for the sound source localization task FIGURE 4.1: Superior Olivary Complex in biology. Green lines represents excitatory connections, while red lines indicates inhibitory connections. Image taken from (Liu et al., 2013). becoming a complex nucleus. For example, contralateral (i.e., opposite side) projections have a different distribution compared to ipsilateral (i.e., same side) projections over the coincidence detector neurons. Moreover, delay lines can adapt themselves depending on the input stimuli. Unlike the MSO, there is not a well-defined model for the LSO. Although it is known that neurons on this nucleus work like a frequency rate subtractor, it is thought that it may work like a frequency coincidence detector, thus having a similar architecture than the MSO. Although there exist many implementations (Liu et al., 2008; Chou et al., 2019; Voutsas and Adamy, 2007), either software or hardware, of the SOC, the MSO, and the LSO models, neuromorphic solutions are becoming popular due to their event-based nature. Following the same design principle that was used during the development of the NAS, in this chapter, an event-based, FPGA-based model of the SOC is proposed with the aim of to continue the development of a more complete neuromorphic auditory sensor. Furthermore, a novel digital implementation of the Time Difference Encoder (TDE) model was carried out and proposed as an alternative of the Jeffress model for the ITD extraction. In this case, instead of having an array of coincidence detector neurons, where in theory only one of them will fire an spike when a coincidence occurs, a single TDE neuron is used to code the ITD in the number of output spikes. Preliminary results showed the viability of this approach for performing the sound source localization task, although future works are needed in order to improve the accuracy. 4.2. Event-based model of the Superior Olivary Complex 81 4.2 Event-based model of the Superior Olivary Complex The Neuromorphic Auditory Sensor (NAS) was implemented by using spikebased building blocks for audio signal processing. Those blocks consist of basic operational blocks that, once combined, are able to perform complex operations in the spike domain. For instance, the addition operation in the events domain is carried out by the Spike Hold&Fire (SHF) module (Jimenez-Fernandez et al., 2010), that imitates a simplified version of a LIF neuron with both excitatory (or first summand) and inhibitory (or second summand) inputs. When this module is syntethized for an FPGA, its resources consumption are negligible, as well as its power consumption. Therefore, this design strategy was also followed for the implementation of the event-based digital model of the Superior Olivary Complex (SOC). As it was previously mentioned, the main goal is to achieve a full model of the ascending auditory pathway, starting with the cochlea and finishing with a spiking neural network which models the auditory cortex. Although a fully VHDL-based model is desired, the SNN would be first deployed in already existing neuromorphic computing platforms, like SpiNNaker (Furber et al., 2014a) or Loihi (Davies et al., 2018), instead of directly to an FPGA due to the SNN model complexity. In addition, such platforms already have development frameworks, thus making some important tasks easier, such as the learning procedure. SpiNNaker FPGA Left NAS Right NAS A V C N A V C N MSO LSO IC ON spikes bus OFF spikes bus FIGURE 4.2: Block diagram of the NSSOC model for FPGA. Taking the biological model shown in Fig. 4.1 as reference, the Neuromorphic Spike-based Superior Olivary Complex (NSSOC) model 1has been designed and implemented as a complement for the NAS design. The main block diagram including both the NAS and the NSSOC is shown in Fig. 4.2. The NAS’ output (ON and OFF events) is directly sent to two modules: 1) the 1The project’s repository is open-source, and it can be found on GitHub https://github.com/ dgutierrezATC/nssoc. 82 Chapter 4. Event-based models for the sound source localization task AVCN, which implements a version of the spherical bushy cells for performing the phase-lock operation, and 2) the LSO, for extracting the ILD. Then, the AVCN output phase-locked events are sent to the MSO module for extracting the ITD. All these modules were implemented by using VHDL language for further deployment into an FPGA-based board. One of the advantages of implementing these models by using the VHDL language is that they can be directly integrated with a NAS model, thus avoiding the need to create a serial interface between the NAS and the NSSOC, e.g., an AER serial interface. Therefore, no additional delays will be added to the precise timing of the events produced by the NAS, which is important for extracting ITDs with high precision. After extracting the binaural cues, the auditory information is sent to the IC, as shown in Fig. 1.14 in Chapter 1, where all the auditory cues are combined for obtaining a spatial map of the sound sources. In the proposed model, the Inferior Colliculus (IC) was implemented as a multi-layer SNN model using PyNN and deployed into the SpiNNaker machine. The communication between the NAS, the NSSOC, and SpiNNaker is carried out through a custom PCB and a modified version of the AER-SpiNNlink Verilog module 2. The following sections describe the implementation of each module in detail, their characterization, as well as an analysis of the performance for different MSO configurations within the NSSOC model. 4.2.1 Implementing the Cochlear nucleus The phase lock effect takes place at the AVCN thanks to the spherical bushy cells. These cells produce an spike when the Sound Pressure Level (SPL) of the input sound is maximum, thus, having the higher displacement in the basilar membrane at that moment. Since this effect is relative to the SPL, it is not dependant on the volume of the input sound. Therefore, for a signal with the same frequency but different amplitude, the resultant phase-locked output spikes will be identical over time. Fig. 4.3 shows an example of this effect, where even though both signals increase their volume, the time difference at which the phaselocked spikes are produced are the same. However, Kandel et al. (Kandel et al., 2000) described this effect more like a periodic response at any time of the spherical bushy cells independently of the SPL instead of an exact response at the maximum value of the SPL, as it was already shown in Fig. 1.16. Therefore, the precise timing and periodicity of the phase-locked events are considered more important than the precise moment in which they are produced. Those two considerations were crucial when designing the phase lock module, since some simplifications were applied. As it was mentioned before, 2The project’s repository is open-source, and it can be found on GitHub https://github.com/ dgutierrezATC/NAS_SpiNNaker_interface. 4.2. Event-based model of the Superior Olivary Complex 83 FIGURE 4.3: Representation of the phase lock effect. Image taken from (Liu et al., 2013). the NAS’ output spikes follow the rate coding, meaning that the higher the value is, the higher the frequency will be (and the lower the ISI will be). Thus, in order to detect the maximum value of the SPL, it should be enough with just detecting the minimum ISI value. Nevertheless, by applying the simplifications, detecting the zero-crossing points of the signal is enough. When using the NAS, this zerocrossing detection process can be made by only detecting the change from ON events (or positive) to OFF events (or negative), since it has both types of outputs. In other models where the output events do not have polarity, as the example shown in Fig. 4.4, the best solution would be the minimum ISI value. (A) Plot showing sample output data from the cochlea model. (B) Plot showing sample output data from the bushy cell neuron. FIGURE 4.4: Example of the phase lock effect. From the digital design point of view, the zero-crossing approach requires less hardware resources compared to the ISI analysis, and the design complexity is also reduced. Therefore, the zero-crossing model was implemented for modelling the spherical bushy cell. The model consists of a register that stores 84 Chapter 4. Event-based models for the sound source localization task the polarity of the last produced event and a comparator that detects the polarity change from positive to negative. The output of the model is a single output that corresponds to the phase-locked event without any polarity. Fig. 4.5a shows the block diagram of the proposed model, and Table 4.1 summarizes its FPGA resources consumption. reg reg = ‘0’ & in = ‘1’ in Input spike polarity out Phase-locked spike (A) Block diagram. 1e7 neg. pos. Input 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Time ( s) 1e7 out. Output (B) Simulation result. FIGURE 4.5: Block diagram and simulation results of the spherical bushy cell model. Firstly, the design was simulated in order to verify its functionality. A basic test, in which a sinusoidal spiking signal was artificially generated and used as input, was carried out. In consonance with the idea, the observed behavior should be to obtain an output event as soon as the first negative event is received at the input. As it can be seen in Fig. 4.5b, the simulation results match the expected behavior, as well as the two considerations which were mentioned before. In addition, the frequency of the input signal can be estimated by measuring the ISI value between two consecutive phase-locked output events, thus facilitating the sound recognition task or pure tones classification (Dominguez-Morales et al., 2016). TABLE 4.1: Hardware resources utilization of the spherical bushy cell model for different FPGA devices. FPGA chip Slice Registers Used / Available Slice LUTs Used / Available XC6SLX150T (Spartan-6) 2/184304 (<0.01%) 0/92152 (0.0%) XC7A75T (Artix-7) 2/94400 (<0.01%) 0/47200 (0.0%) XC7K480T (Kintex-7) 2/597200 (<0.01%) 0/298600 (0.0%) Secondly, four real-time tests were performed in order to analyze the impact of the spherical bushy cell module within the NAS in terms of number of output events and the NAS’ response in the frequency domain. The tests were organized as follows: 1) white noise sounds with the same volume (power equal to 0 dB); 2) white noise sounds with three different volumes (power equal to -6, 0, and 6 dB); 3) a set of pure tones with the same volume (amplitude equal to 1); and 4) a set of pure tones with three different volumes (amplitude equal 0.5, 1.0, 1.5, 2.0, and 4.2. Event-based model of the Superior Olivary Complex 85 2.5). The sounds were one second long, and the sample frequency was 48 MHz. The frequencies for the pure tones test were set to 100, 250, 500, 1000, 2000, 5000, 10000, 15000, 20000, and 25000 Hz. A 64-frequency-channel, cascade, mono NAS was generated using OpenNAS, and a set of 64 spherical bushy cells were also instantiated and connected to the NAS’ output. This set of spherical bushy cell would create the AVCN module, which only contains as many spherical bushy cell modules as NAS frequency channels. The I2S-based input was used for sending the sounds from the computer to the FPGA-based board in real time. The output events were collected using the USBAERmini2 board and a Matlab script3for automating the process.                   (A) 500 Hz, amplitude 0.5, histogram. 0 20 40 60 80 Bin (10000 s width) 0 20 40 60 80 100 120 Address Sonogram 0 25 50 75 100 125 150 175 No. of spikes (B) 500 Hz, amplitude 0.5, sonogram.                   (C) 500 Hz, amplitude 1.5, histogram. 0 20 40 60 80 Bin (10000 s width) 0 20 40 60 80 100 120 Address Sonogram 0 50 100 150 200 250 300 350 No. of spikes (D) 500 Hz, amplitude 1.5, sonogram. FIGURE 4.6: Histograms and sonograms for a 500 Hz pure tone with 0.5 and 1.5 of amplitude using a NAS without phase-lock. The experiment was carried out twice by enabling and disabling the AVCN module for comparison purposes. Next, a few representative results among all 3https://github.com/dgutierrezATC/GenericSeqMon 86 Chapter 4. Event-based models for the sound source localization task the obtained outputs are shown. The full set of output results can be found on GitHub 4. Fig. 4.6 shows the results obtained from the test in which the AVCN module was disabled, and a pure tone of 500 Hz was played at two different volumes (0.5 and 1.5). As it can be seen, both histograms have the main component around the same frequency. Fig. 4.7a presents a noise peak placed at high frequencies due to the low volume and the inherent error of the Spike-based Band-Pass Filter (SBPF) (Jimenez-Fernandez et al., 2017). Nevertheless, this error can be reduced while increasing the signal amplitude, as shown in Fig. 4.7c. Additionally, the number of events was different in both cases, having a maximum value of 17500 for amplitude 0.5 and 35000 for amplitude 1.5. These results matched the expected behavior of the NAS, which was already analyzed in (Jimenez-Fernandez et al., 2017; Dominguez-Morales et al., 2016).                 (A) 500 Hz, amplitude 0.5, histogram. 0 20 40 60 80 Bin (10000 s width) 0 20 40 60 80 100 120 Address Sonogram 0 20 40 60 80 100 120 No. of spikes (B) 500 Hz, amplitude 0.5, sonogram.                 (C) 500 Hz, amplitude 1.5, histogram. 0 20 40 60 80 Bin (10000 s width) 0 20 40 60 80 100 120 Address Sonogram 0 20 40 60 80 100 120 No. of spikes (D) 500 Hz, amplitude 1.5, sonogram. FIGURE 4.7: Histograms and sonograms for a 500 Hz pure tone with 0.5 and 1.5 of amplitude using a NAS with the phase-lock module. 4In https://github.com/dgutierrezATC/nssoc, go to Examples, NAS_AVCN, NAS_I2S_64ch_mono_12att_monitor, realtest, GenericSeqMon, dataset 4.2. Event-based model of the Superior Olivary Complex 87 After the first experiment, which established the ground truth, the AVCN module was enabled and the whole experiment was carried out again. Fig. 4.7d shows the obtained results from the same cases shown in Fig. 4.6. Many differences can be appreciated from the results obtained in this experiment. For instance, there is only one peak in the histogram. However, this peak is not centered in the addresses that correspond to the pure tone frequency, but it is placed at the high frequency addresses. On the contrary, the addresses corresponding to the 500 Hz frequency showed an uniform response in terms of output events instead of a peak, as can be seen in Fig. 4.7a and Fig. 4.7c. This effect is produced by the Spike-based Band-Pass Filter (SBPF) implementation since the band-pass filter is implemented by using two Spikebased Low-Pass Filter (SLPF) in serial for the NAS cascade architecture. Therefore, the achieved band is not as precise as desired, thus, having several frequency channels responding equally to the input sound (Jimenez-Fernandez et al., 2010; Jimenez-Fernandez et al., 2017). This way, the peak can be considered as noise, and its high value is also due to the tuning process of the SBPF, where the error is higher while the SBPF’s middle frequency is also higher. Ideally, the output should present some activity around the addresses associated with the input sound and no activity in the rest. (A) 500 Hz, amplitude 1.5, without phase-lock. (B) 500 Hz, amplitude 1.5, with phase-lock. FIGURE 4.8: Examples of recordings using NAS without and with phaselock. Furthermore, the volume of the input signal almost does not have any effect on the NAS output response. The maximum number of output events are practically identical. Nevertheless, two effects can be observed. Firstly, the flat part, i.e., the addresses with the same activity in the histogram, is shorter for louder volumes. Secondly, the peak’s shape is softer when the volume is lower, and more abrupt when the volume is higher. Related to the overall number of output spikes produced by the NAS, Fig. 4.8 shows the spikegram of both studied cases. Although the reduction of the 94 Chapter 4. Event-based models for the sound source localization task added. This overlap increases the coincidence detection accuracy by producing, in some cases, two coincidence spikes at the output. left right 0 1 2 3 4 5 6 7 Outputs 8 9 10 11 12 13 14 0 100000 200000 300000 400000 500000 600000 Time ( s) 15 FIGURE 4.14: Example of the ITD network when using a sweep. Therefore, the detection time bin of a single coincidence detector neuron depends on the global detection time of the network, as well as the total number of coincidence detector neurons deployed, and the detection overlap. It can be defined as in Equation 4.1. detection_time_bin =MAX_DETECTION_TIME NUM_NEURONS +DETECTION_OVERLAP (4.1) In the same way, the transmission time of a single delay line is calculated also based on the global detection time of the network, as well as the total number of coincidence detector neurons deployed, and it is proportional to the index of that specific transmission line. For instance, the first delay line (with index equal to 0) has no delay, the second one (with index equal to 1) has one time the 4.2. Event-based model of the Superior Olivary Complex 95 MAX_DETECTION_TIME NUM_NEURONS , and so on. Thus, the transmission time of the i-th delay line can be defined as in Equation 4.2. transmission_timei=MAX_DETECTION_TIME NUM_NEURONS ×i(4.2) A behavioral simulation was carried out for validating the module. The test consisted of the generation of an input spike pair where the time difference between them was increasing over time. Therefore, a sound source moving from left to right with respect to the reference can be simulated. The NUM_NEURONS parameter was set to 16, the MAX_DETECTION_TIME parameter was set to 700 µs, and the DETECTION_OVERLAP parameter was set to 5µs. The generated input stimuli were organized in bursts of five spike pairs with one millisecond between pair generations. Firstly, the time difference between the left and right stimulus was set to 700 µs, meaning that the sound source was completely placed at the left with respect the reference, creating a 90 degree angle. Then, for each new spike burst, the time difference value was decreased by 20 µs, simulating the movement of the sound source towards the center with respect to the reference. This process was repeated while the time difference was higher than zero, meaning that the sound source was not placed in front of the reference, where the time difference is zero. Secondly, once the simulated sound source was placed in front of the reference, with an angle of zero degrees, the time difference was increased for each burst again by 20 microseconds but producing first the right spike and then the left spike, simulating that the sound source was being moved towards the right ear. The simulation was stopped when the time difference reached 700 µs, meaning the sound source was then placed at a 90 degree angle on the right position with respect to the reference. Results from the simulation are shown in Fig. 4.14. Two main effects can be clearly observed. On the one hand, it can be seen that the burst’s width decreases when the time differences are closer to zero since the coincidence is detected some microseconds earlier in these cases compared to high time differences. On the other hand, there are some cases where two consecutive neurons detected a coincidence produced by the same input stimulus pair due to the overlapping configuration. In general, the network worked according to the expected output, proving that the model was successfully implemented. A resources consumption report of this model was generated for different FPGA chips and collected in Table 4.4. For a single ITD detection network, 2.89% of the resources are needed in the worst case, meaning that a total of 34 ITD detection networks could be deployed without taking into account the NAS module. Since the ITD extraction is valid for low frequency channels, a set of ten 96 Chapter 4. Event-based models for the sound source localization task TABLE 4.4: Hardware resources utilization of the ITD network model with a global detection time of 700 µs and 16 coincidence detection neurons for different FPGA devices. FPGA chip Slice Registers Used / Available Slice LUTs Used / Available XC6SLX150T (Spartan-6) 788/184304 (0.42%) 1294/92152 (1.40%) XC7A75T (Artix-7) 764/94400 (0.81%) 1364/47200 (2.89%) XC7K480T (Kintex-7) 764/597200 (0.12%) 1364/298600 (0.45%) to fifteen ITD detection networks could be enough, thus still having free resources for future implementations. 4.2.2.5 Medial Superior Olive model Based on the biological principles of the MSO introduced in Section 1, it can be modelled as a set of ITD detection networks tonotopically organized and connected one-to-one to the phase-locked NAS output spikes from lowfrequency channels. In this manner, a MSO module was implemented for automatically deploying as many ITD detection networks as specified by the generic parameters, thus facilitating the modules’ configuration and the output events collection. As input lines, the module has four times the value of the generic parameter NUM_FREQ_CH, since for each frequency channel it will have inputs for both left and right, as well as positive and negative spikes for each side. As output, the module has an AER interface, thus allowing the communication with an external device or with other modules within the same FPGA. The model was composed by layers, where each layer consisted of two spherical bushy cell modules (SBC), an ITD detection network module, and a local events monitor module, as shown in Fig. 4.15. The number of layers can be defined by the generic parameter NUM_FREQ_CH, that indicates the number of NAS’ frequency channels from which the events will be received. Two generic parameters, START_FREQ_CH and END_FREQ_CH, were also needed in order to set both the initial and the final NAS frequency channels that are going to be used. In addition, these values were also used to configure the channel’s address for each local monitor. In the same way that the ITD detection network module, the networks’ configuration were carried out through the generic parameters NUM_NET_NEURONS,MAX_DETECTION_TIME, and DETECTION_OVERLAPING. These parameters are shared by all the ITD detection networks inside the MSO module, and they cannot be modified in 4.2. Event-based model of the Superior Olivary Complex 97 ITD network Local monitor SBC SBC Global monitor & AER interface ... ... on off on off Spikes from left freq. channel i Spikes from right freq. channel i left right 0 1 2 n-1 0 1 m-1 To SOC AER merger req data ack 16 16 16 16 Sinlge spike line Control line Data bus FIGURE 4.15: Block diagram of the proposed MSO model. real-time. This feature could be implemented by adding a dedicated Random Access Memory (RAM) memory which would be modified by an external controller, and it will be implemented in the next version. For collecting the events from each ITD detection network, a spikes monitor was used. In a study carried out by Cerezuela-Escudero et al. in (Cerezuela- Escudero et al., 2013), where different approaches for implementing an event monitor were compared, the global monitor was identified as a potential bottleneck point when the output spike rate was high enough. In addition, both the organization and the number of spike lines in the module which is being monitored were also considered as a critical aspect when designing a spike monitor. The work compared a global spikes monitor, where all the spikes were sent to a single monitor, with a distributed spikes monitor, where the spikes where divided in groups and, for each group, a local monitor was used. This way, the workload was distributed, thus reducing the bottleneck problem of the global monitor output AER interface. Results showed that the distributed monitor was more efficient in terms of events loss, as well as in terms of FPGA resources consumption. Therefore, a distributed events monitor, composed of several local monitors and a global monitor, was used in this work. The block diagram of the local monitor module is shown in Fig. 4.16. First, an event encoder was used for identifying the input spike and labelling it with a number between [0, NUM_NET_NEURONS −1]. The event encoder module was designed in such a way that only one event is encoded at each clock cycle. If two or more events arrive at the same time, the event with the lowest ID (i.e., closer to zero) will be encoded. Then, the encoded events are stored in a FIFO memory. The reading process is controlled by the arbiter module in the global monitor module. When the 98 Chapter 4. Event-based models for the sound source localization task Events encoder FIFO Mask log2(n) 0 1 n-1 ... out new event data_in data_out no_empty log2(n) 16 Output spikes from ITD network Data available signal to global monitor arbiter w r Masked event data Read data signal from arbiter Sinlge spike line Control line Data bus FIGURE 4.16: Block diagram of the MSO local events monitor. arbiter enables the read signal, a data from the FIFO is read, and a mask is applied to the read data. This mask module concatenates the already stored neuron label with: • Event polarity, that is always zero. • Associated NAS frequency channel, determined by the ID of ITD detection network within the MSO module and both START_FREQ_CH and END_FREQ_CH parameters. • Left/right ear, which is always zero since only one MSO nucleus is instantiated. • One bit fixed to ’0’ for indicating that the event was produced by the MSO module. • One bit fixed to ’1’ for indicating that the event was produced by the SOC module and not by the NAS. The resultant masked event is then sent to the global monitor module, where it is stored in another FIFO memory. The arbiter’s output is used as selector for a multiplexer, which selects the correct masked event data to be stored in the FIFO memory. Therefore, this memory will contain data from all the ITD detection networks. Finally, the data stored in the FIFO memory is sent out through the AER out interface to either another FPGA module or an external device. The block diagram of the global monitor module is shown in Fig. 4.17. As it was done before, a resources consumption report of this model was generated but only for the Artix-7 FPGA chip, which will be used for the realtime experiments. Table 4.5 shows the obtained results for a MSO module with 10 ITD detection networks (connected to the NAS frequency channels between 25 and 34), 16 coincidence detection neurons for each ITD network, and a global detection time of 700 µs with an overlapping time of 5 µs. As it can be seen, the MSO module almost needs 30% of the FPGA’s Look-Up Tables (LUTs) due to the multiple instantiations of both the delay lines and the coincidence detector 4.2. Event-based model of the Superior Olivary Complex 99 Arbiter Mux Encoder FIFO AER out interface ... ... 16 16 16 16 m log2(m) 16 16 Data available signal from local monitors REQ DATA ACK Sinlge spike line Control line Data bus 0 1 m-1 r w data_in no_empty data_out grant in out new 0 1 m-1 sel out busy data_in new Masked events from ITD networks FIGURE 4.17: Block diagram of the MSO global events monitor. neurons. This means that for a more complex MSO model, an FPGA with much more resources would be needed, taking also into account that there should be at least 50% of the resources available for the NAS model. TABLE 4.5: Hardware resources utilization of the MSO model with a global detection time of 700 µs, 10 ITD networks and 16 coincidence detection neurons per ITD network for the Artix-7 FPGA chip. Module name Slice Registers Used / Available Slice LUTs Used / Available Left AVCN 20/94400 (0.02%) 0/47200 (0.0%) Right AVCN 20/94400 (0.02%) 0/47200 (0.0%) MSO 7640/94400 (8.09%) 13649/47200 (28.92%) MSO monitor 229/94400 (0.24%) 331/47200 (0.70%) Once again, a behavioral simulation was performed in order to validate the implemented module. In this case, the input stimulus was not generated manually due to its complexity. Instead, a NAS model was used for generating an events file. Firstly, a 64-frequency channels, stereo NAS with I2S-based input interface was generated. Then, a virtual room with a pair of microphones and a sound source was created by using the Room Impulse Response (RIR) generator (Habets, 2006) Python package. The room’s dimensions were 10 ×10 ×4 meters (x, y, and z). The microphone pair (subcardioid type, oriented as the human ears, with a separation of 0.3 meters) was placed at (5, 3, 2), and the sound source was placed at (5, 4.5, 2). The sound velocity was set to 340 m/s, and the reverberation time was set to 0.2 seconds. Finally, the sample frequency was set to 48000 Hz, and a pure tone of 1000 Hz with 0.5 seconds of duration was played. The output audio file, generated by the RIR generator tool, was then used as input for the NAS. The NAS was deployed into the ZTEX 2.13 board 100 Chapter 4. Event-based models for the sound source localization task along with two custom boards: a baseboard (see Appendix C.2), and an audio input board (see Appendix C.1) for interfacing the baseboard with the computer audio interface. The audio file was played from the computer with a 80% of volume, and the output events were collected into an .aedat file by using the USBAERmini2 board and jAER (Delbruck, 2007). Next, the events file was loaded by using a Python script and pyNAVIS tool (Dominguez-Morales et al., 2021b). After checking that all the values were correct according to the settings, two lists were generated: one containing the addresses and another one containing the timestamps. Those lists were used as input files for the VHDL simulation, and the output events generated by the MSO model were also saved as a text file with a specific format. Finally, the results files were loaded by using pyNAVIS5, and a set of plots were generated. These plots are shown in Fig. 4.18. On the top left, the MSO spikegram is shown. This plot shows the raster plot, or the raw output events, produced by the MSO module. The x-axis represents the neurons’ IDs, the y-axis represents the time, and the z-axis represents the NAS frequency channels, being channel 0 the one with the highest frequency and channel 63 the one with the lowest frequency. Since the sound source was placed in front of the microphone pair, the main activity was produced by neurons with IDs between 6 and 10 along the simulation. However, some activity was generated by other neurons which do not correspond to the correct localization due to the noise added when using a real-time setup for generating the recording. On the top right, the MSO histogram is presented. The histogram shows the same information that the spikegram but without taking into account the time. Instead, the total number of events generated by each neuron in each frequency channel is shown. As it was mentioned before, the main activity was found in the neurons placed at the middle. Nevertheless, not all the frequency channels presented activity in those neurons. Frequency channels with a middle frequency close to the pure tone frequency had a better response compared to those frequency channels that have lower middle frequencies. On the bottom left, the MSO heatmap is shown. In this plot, the same information of the histogram is shown but just using a two dimensions color map: the frequency channels and the neuron IDs. The number of output events is represented by a color code, where the meaning of each color is determined by the color bar placed on the left. In addition, the color assigned to each position is calculated relatively to the maximum value. Therefore, this view facilitates the identification of the main activity within the MSO model and gives an idea on how sparse was the model response to the input stimulus. Finally, on the bottom right, the position estimation extracted from the MSO output is plotted. For generating this plot, a software version of the coincidence 5The functionalities needed for loading, verifying, postprocessing and plotting the MSO model information were implemented as part of this thesis. 4.2. Event-based model of the Superior Olivary Complex 101                       (A) MSO spikegram.                        (B) MSO histogram.                                                                                                                                                                                                    (C) MSO heatmap.                              (D) MSO localization estimation. FIGURE 4.18: Results from a behavioral simulation of a MSO model when using a 1000 Hz pure tone, placed in front of the reference, with a distance of 1.5 meters. The plots were generated using pyNAVIS. counters from the Jeffress model was used. First, a time bin is defined for integrating the events of each coincidence detector neuron. Then, the events are counted, and the neuron with the maximum fired events is selected as the winner. Finally, the ID of the winner neuron is converted to an angle, being this angle the estimated position of the sound source in the horizontal plane with respect to the microphone pair, as shown in Equation 4.3. The value of max_angle_accuracy can be obtained according to Equation 4.4, and it directly depends on the model’s architecture. estimated_angle =max_angle_accuracy ×neuron_ID (4.3) 102 Chapter 4. Event-based models for the sound source localization task In this case, it can be seen that the position of the sound source was estimated in a range of ±5 degrees with respect to the reference, which is in front of it. Since there was an even number of coincidence detector neurons, the winner neuron was oscillating between two consecutive positions. The big oscillations observed at the end of the plot correspond to the end of the input stimulus, where the MSO model did not receive any sound, thus producing a random output due to the existing background noise. The accuracy of the model directly depends on the number of coincidence detector neurons and it could be measured from this plot. However, the accuracy also depends on many other factors, such as the room shape and its dimensions, the room’s reverberation, the input sound, the distance between microphones and the sound source, etc. Therefore, the accuracy study can be considered application-specific measurements, thus not being carried out in this experiment. This process was not integrated along with the MSO model since the time bin depends on the final application, thus needing to be configured or even adapted in real time. In addition, most of the times the MSO events will be sent to a neuromorphic computing platform, like SpiNNaker, for further processing like sensory integration, multiple sound sources localization, or auditory attention. Nevertheless, the coincidence counter module was implemented and tested using VHDL for future works. 4.2.3 Lateral Superior Olive According to the block diagram showed in Fig. 4.2, the next nucleus to be implemented would be the LSO for the ILD extraction. A reduced eventbased model of the LSO was already implemented by Cerezuela-Escudero et al. in (Cerezuela-Escudero et al., 2018). In that work, the SHF model was used and combined in groups of four to create a new module: the spike firing rate subtractor. This module is able to produce a spike stream whose firing rate is proportional to the rate difference of two input spike streams. Therefore, the ILD is encoded in the output spike rate. Details about the implementation can be also found in (Jiménez Fernández, 2010). The basic module has four inputs: two of them are for the spikes coming from the left cochlea (positive and negative spikes), being considered as the excitatory input; and the other two inputs are for the spikes coming from the right cochlea, being considered as the inhibitory input. Since the NAS model already implements positive and negative spikes, a model of the Medial Nucleus of the Trapezoid Body (MNTB) nucleus was not needed. Therefore, according to the design strategy used for the implementation of the MSO module, the LSO module would be implemented for automatically generating a set of SHF modules. The number of ILD extraction units would be equal to the number of frequency channels specified in the generic parameter 4.2. Event-based model of the Superior Olivary Complex 103 NUM_FREQ_CHANNELS that indicates the number of frequency channels from which the ILD will be extracted. In addition, the module would also have two generic parameters for indicating both the first and last frequency channel addresses (START_FREQ_CH and END_FREQ_CH). Finally, the number of input signals would be four times the value of NUM_FREQ_CHANNELS since, for each ILD extraction unit, two inputs from the left cochlea and two inputs from the right cochlea are needed. The number of output signals would be twice the value of NUM_FREQ_CHANNELS since, for each ILD extraction unit, both positive and negative output signals are available. FIGURE 4.19: ILD (encoded as an event-rate) measured over time obtained from the input data that generates the left and right NAS when the sound source is placed at 45° from the head. Figure taken from (Cerezuela- Escudero et al., 2018). Fig. 4.19 shows an example of the LSO model response. A sound was played with an angle of 45 degrees with respect to the head. As it can be seen, the output firing rate of input A was higher than the output firing rate of input B since the sound source was placed closer to the input A, thus receiving the sound earlier and with more power than to the input B. In fact, the event rate of input A is 2.5 ×104events/sec, while the event rate of input B is 2 ×104events/sec, thus obtaining an estimated ILD of 0.5 ×104events/sec. According to the biological principles described in Chapter 1, the ILD works better with high frequency sounds. The ILD estimation can be used for performing the sound source localization tasks, but it is also used for helping the MSO to better determine the position of the sound source by correcting the ambiguity that the high frequency sounds produce at the MSO (Dávila-Chacón et al., 2018). Nevertheless, the latest research about the LSO and the ILD extraction suggests that the LSO’s neurons are temporal differentiators rather than integrators (Franken et al., 2018), meaning that those neurons not only take into 110 Chapter 4. Event-based models for the sound source localization task the same and the MSO module within the SOC module was modified. In the first configuration (identified as "model 1"), 4 ITD detection networks, connected to frequency channels from the 25th to the 29th, were used, with 32 coincidence detector neurons for each network. In the second configuration (identified as "model 2"), 10 ITD detection networks, connected to frequency channels from the 25th to the 34th, were used, with 16 coincidence detector neurons for each network. In both cases, the global detection time was set to 700 µs, with an overlapping time of 10 µs. The middle frequencies for each channel, from channel 25 to 24, were: 1366.22, 1222.49, 1093.88, 978.80, 875.82, 783.68, 701.24, 627.46, 561.45, and 502.38, respectively and expressed in Hz. In total, 540 .aedat files were generated. Next subsections will show some representative examples for comparing and analyzing the impact of different models’ configurations over the three aspects already mentioned: the configuration of the model, the frequency, and the distance. A full analysis, taking into account all the files, was not carried out due to the high number of samples. In addition, the analysis would depend on the final application, the input stimuli, and the model configuration. For this reason, this qualitative analysis was performed. Nevertheless, an individual analysis for each .aedat file was automatically generated using pyNAVIS tool, and they are available on the GitHub project’s repository. 4.2.6.1 How does the model affect? One of the most important decisions before starting a new application is the model’s configuration to be used. This decision directly depends on the application’s nature, since the parameters of the model should match with the application’s requirements. For example, if a high precision is needed, the number of coincidence detector neurons for each ITD detection network should increase, thus decreasing the number of ITD detection networks in order to balance the FPGA resources consumption. This also implies to the model for being more selective in the frequency range from where the ITD wants to be extracted. On the other hand, a MSO model with fewer neurons per network but with more networks would allow covering a wider frequency range, thus being useful for detecting multiple sound sources in different frequencies. In this case, the same test file was used for both models. It consisted of a pure tone of 750 Hz, played for 0.5 seconds, placed at position 5 over 9 (i.e., in front of the reference with an angle of 0 degrees) with a distance of 0.5 meters. Then, for each result file, three different plots were generated: the MSO spikegram, showing the raw output events of the model, the MSO heatmap, showing the activity of each neuron for each network, and the MSO position estimation, which gives an idea of the accuracy of the model. These six plots are shown in Fig. 4.24. It can be observed in the spikegram that model 1 produces almost the same number of spikes compared to model 2. About the events’ distribution, the 4.2. Event-based model of the Superior Olivary Complex 111                        (A) MSO spikegram from model 1.                       (B) MSO spikegram from model 2.                                                                                                                                                                        (C) MSO heatmap from model 1.                                                                                                                                                                                                    (D) MSO heatmap from model 2.                                              (E) MSO localization estimation from model 1.                              (F) MSO localization estimation from model 2. FIGURE 4.24: Comparison between different configurations of the MSO. 112 Chapter 4. Event-based models for the sound source localization task output events from model 1 are more clustered around neurons 15 and 16 (which are the middle neurons, corresponding to the position of the sound source), while model 2 produced a more sparse response. Those events which are far from the main cluster could be considered as noise, and it is a direct consequence of the model’s configuration. Therefore, it can be confirmed that the model’s accuracy directly depends on the number of neurons, as it was expected according to the state-of-the-art. Furthermore, the heatmaps show almost identical global behavior, presenting both models a cone-like response. The highest activity was found on the peak of the cone, starting to be more sparse while decreasing the middle frequency of the frequency channels. Finally, the sound source position estimation shows that both models are able to determine, with the same accuracy, the real position of the sound source. The oscillations in the estimation were produced because the number of neurons was not an odd number, thus being both positions considered as winners. Summarizing, the model’s parameters have a crucial influence on its response, and they should be chosen according to the final application. The global response of the model would be almost the same, and the final decision would depend on the preliminary results obtained from the first tests carried out by the user. From that point, the model could be finely tuned to match the application’s requirements. In our case, model 2 was selected for performing the rest of the tests since it covers more frequency channels and, for this application, the accuracy is not relevant. 4.2.6.2 How does the frequency affect? As it was mentioned in Section 1.3.1.2.2, the MSO works better for low frequencies (in the range of 300 Hz to 1200 Hz). From that frequency to higher frequencies, the period of the sound becomes equal or lower than the coincidence detection time, thus producing ambiguities in the coincidence detection that can be corrected later by the LSO. This situation could be also improved by decreasing the coincidence detection time in the ITD detection networks associated to higher frequency channels. Frequently, the tuning process of the model’s parameters is performed by mean of a training procedure (Liu et al., 2013). However, since the learning process implies the development of the learning algorithm within the system, it was not implemented in this work, and the configuration process was carried out by hand. To this end, two different situations were then analyzed in order to verify the response of model 2 to this kind of situation where the frequency of the input sound has either low or high frequency. The first analyzed case was the NAC’s output response for a pure tone of 500 Hz, placed at position 3 over 9 (30 degrees approximately from the very right position) with a distance of 0.5 meters with respect to the reference. The second analyzed case was the NAC’s 4.2. Event-based model of the Superior Olivary Complex 113 output response for a pure tone of 1500 Hz, also placed at position 3 over 9 with a distance of 0.5 meters with respect to the reference. The resultant plots are shown in Fig. 4.25. From the spikegram, it can be already observed that the model’s response to the high frequency sound (Fig. 4.25b) does not offer any localization information since there is not any main cluster of points. On the contrary, the spikegram of the low frequency sound clearly shows the sound position through the model’s response, concentrating all the produced events around neuron 3. Moreover, the heatmaps offer a better view about the behavior of the models. For the 500 Hz pure tone, the system was able to determine that the position of the sound source was more to the right, while for the 1500 Hz pure tone, the system was not able to determine any region of potential localization. In fact, three different high activity points can be identified, meaning that the ambiguity was too high, probably because the distance between the microphones. Taking a look at the localization estimation plots, it can be seen that the estimation is highly precise for the case showed in Fig. 4.25e. The plot shows that the sound source is placed at 60 degrees to the right with respect to the reference, that matches with the expected result. The final oscillations were produced because the noise at the end of the audio, thus it should not be taken into account. However, if we take a look at the localization estimation for the 1500 Hz sound case, the result is a fully oscillatory behavior without any accurate estimation. This effect could be produced due to the distance between microphones does not match the global coincidence detection time. Theoretically, the MSO’s response should be acceptable with sounds up to 1500 Hz, although for this configuration that limit was lower. Therefore, it can be said that the frequency of the input sound is a critical aspect to take into account for potential applications, like speaker localization or auditory attention. The maximum accuracy of the system, expressed in degrees, is determined just by the number of coincidence detector neurons, and it can be described as in Equation 4.4. maxaccuracy =180 NUM_NEURONS (4.4) The accuracy is typically measured for the front side with respect to the reference since the back side follows the same principles than the front side but with the difference that the sound’s power is lower due to the ear’s shape. Another factor that affects on the accuracy is the distance between the reference and the sound source, and it will be analyzed next. 114 Chapter 4. Event-based models for the sound source localization task                       (A) Output spikegram for 500 Hz input sound.                       (B) Output spikegram for 1500 Hz input sound.                                                                                                                                                                                                       (C) Output heatmap for 500 Hz input sound.                                                                                                                                                                                                    (D) Output heatmap for 1500 Hz input sound.                              (E) Localization estimation for 500 Hz input sound.                              (F) Localization estimation for 1500 Hz input sound. FIGURE 4.25: Frequency comparison using model 2. 4.2. Event-based model of the Superior Olivary Complex 115 4.2.6.3 How does the distance affect? The distance at which a source is producing the sound is important for many reasons. For instance, the sound wave loses some power while it is being propagated through the air, as well as it is expanded in all the directions. Those effects are especially notable in low-frequency sounds, thus coinciding with the frequency range in which the MSO works better. This situation could lead to situations in which the model becomes less accurate due to the ambiguity. According to (Risoud et al., 2018), the ITD and ILD provide precise localization in the azimuthal plane, with the exception of what is known as the “cone of confusion”. For sounds coming from the circumference of this cone, the axis of which is the interauricular line, there are no time or level differences, leading to confusing perceptual coordinates: the subject is unable to tell whether the sound is coming from the front or from behind, above or below, or from anywhere else along the circumference shown in Fig. 4.26. For any sound source with coordinates ∆,α,θthere is a mirror-image position (∆, 180, −α,−θ)with similar ITD and ILD. FIGURE 4.26: Concept of cone of confusion. Figure taken from (Risoud et al., 2018). The cone of confusion is critical when analyzing the accuracy of an application and for tuning the localization estimation in robotics applications. There are some metrics that measure the ambiguity or confusion, such as the one presented by Fischer et al. in (Fischer et al., 2020). However, the concept of accuracy could vary depending on the final application, and it should be first defined and then measured. In addition, there are dynamic cues that can reduce ambiguity. According to (Wallach, 1939) and (Risoud et al., 2018), by moving the head (or, in animals, the ears) extra binaural and spectral cues are introduced, thus enhancing localization. By leaning the head (and thus the vertical interaural 116 Chapter 4. Event-based models for the sound source localization task axis) or turning it, the amplitude and phase of the sound waves reaching either ear are altered, providing dynamic binaural cues. Head movements also provide an accumulation of Head-Related Transfer Functions (HRTFs) and the creation of multiple cone of confusions, refining localization by combining the information and thus constraining probability of a sound source location to a small number of possible sources. Risoud et al. concludes in (Risoud et al., 2018) that, like for other sensory stimuli, auditory perceptual disambiguation also involves integrating multiple other sensory input, notably visual. Once a sound has been located as coming from an specific area and a specific distance, visual data help fix the position. Moreover, prior knowledge concerning the location of the sound source helps determine its present location. It is noteworthy that images take precedence over sounds, in the “proximity-image effect” described by Gardner in 1968 (Gardner, 1968): if some visual clue is available, the sound source will be located accordingly, even if mistakenly. Again, two different situations were analyzed for verifying how the cone of confusion affects to the proposed model. For the first case, the NAC’s output response for a pure tone of 500 Hz, placed at the position 4 over 9 (65 degrees approximately from the very right position) with a distance of 0.5 meters with respect to the reference was analyzed. Then, for the second case, the NAC’s output response for the same pure tone but with a distance of 3.0 meters with respect to the reference was analyzed. The resultant plots are shown in Fig. 4.27. Obtained results show the effect of the cone of confusion with the increment of the distance. Comparing both histograms, the activity of the model when the sound source is placed at 0.5 meters is more focused on the winner neuron (neuron 5), while the activity when the sound is placed at 3.0 meters is more distributed over the neuron 5 and its surroundings. In fact, the heatmaps clearly show this behavior. Heatmap associated to a closer distance between the sound source and the reference shows that the winner neuron is the same for all the ITD detection networks, as well as that the difference of the activity with respect to the surrounding neurons is notably higher. On the contrary, the heatmap associated to the other case presents a sparse activity distribution around neurons 5 and 6, where the range of neurons with notable activity can be established between neurons 3 and 8. Although in this case the ambiguity is not high enough for leading the system to have a clear confusion, it can be seen how the distance directly affects the accuracy of the estimation of the sound source position. This way, it can be observed in Fig. 4.27e that the localization estimation was 100% focused on neuron 5 (62 degrees with respect to the right), without taking into account the last part of the audio. However, for the case where the distance is 3.0 meters, the estimation presents a slightly noisy behavior between four different positions. 4.2. Event-based model of the Superior Olivary Complex 117                          (A) Histogram for a distance of 0.5 meters.                         (B) Histogram for a distance of 3.0 meters.                                                                                                                                                                                                        (C) Heatmap for a distance of 0.5 meters.                                                                                                                                                                                                     (D) Heatmap for a distance of 3.0 meters.                              (E) Localization estimation for a distance of 0.5 meters.                               (F) Localization estimation for a distance of 3.0 meters. FIGURE 4.27: Effect of the cone of confusion for model 2. 118 Chapter 4. Event-based models for the sound source localization task TABLE 4.7: Comparison of the SOC model implementations. Paper reference Model/s Platform Configuration Real-time (Lazzaro and Mead, 1989b) MSO ASIC Manually No (Liu et al., 2013) MSO Software Manually No (Glackin et al., 2010) MSO FPGA STDP Yes (Dávila-Chacón et al., 2018) MSO, LSO, IC Software Bayesian inference Yes This work MSO FPGA Manually Yes Applied to a robotic platform, this oscillatory behavior could drive a situation in which the oscillation amplitude increases over time with the robot movement, thus finally producing a failure. As it was mentioned before, this problem could be solved by integrating visual information for improving the final estimation and reducing the oscillation. In addition, the localization estimation is not fully performed in the MSO, but it is complemented in the Inferior Colliculus (IC), where a final sound source localization is performed. After that, this auditory information is sent to the Superior Colliculus (SC), where the information is combined with the visual information for taking the final decision. This whole process, although it is complex, would be desired in human-robot interaction applications for improving the user’s experience. 4.2.7 Conclusion In this section, an event-based model of the Superior Olivary Complex (SOC) for performing sound source localization tasks has been presented. This model is part of the development of a completely digital, event-based neuromorphic hearing sense, starting with the NAS and ending with the auditory cortex. By following the same strategy as used in the NAS design, in which spike building blocks (Jimenez-Fernandez et al., 2010) were used, the model was able to successfully extract the ITD from the input stimuli. In addition, the model was integrated with the NAS, and a set of real-time experiments were carried out for validating the proposed implementation. Table 4.7 compare this work with the works used as reference. Some features, like the implemented models, if it was implemented either in software or hardware, as well as if the model’s parameters are tuned by hand or by applying some learning techniques, were qualitatively compared. Results obtained from the experiments showed that the model implements the features described in Section 1.3.1.2 and in Section 4.1, like the effect of the distance and the effect of the frequency in the MSO model. Further analysis are required for measuring the accuracy of the model, although preliminary results 4.3. Alternatives to the Jeffress model: The Time Difference Encoder 119 indicates the system is able to determine the position of the sound source very precisely with an error of a few degrees. The angular resolution of the model could be improved after integrating the output information of the MSO model with the output information of the LSO model. By combining those modules’ outputs with a biologically-inspired model of the IC, this sound source localization system could be used for solving complex situations, like auditory attention tasks when multiple sound sources are present (Chou et al., 2019). Finally, the NAC model is suitable to be integrated in many robotic platforms due to its VHDL-based nature, thus allowing it fast integration within any FPGA-based board 9. 4.3 Alternatives to the Jeffress model: The Time Difference Encoder A recently proposed model which computes temporal dependencies in SNNs is the spiking Elementary Motion Detector (sEMD) (Milde et al., 2018). In that work, the sEMD model consisted of two parts: an event-based vision sensor as input and the TDE as sensory pre-processing unit. The TDE unit translates the time difference between two events into a burst of output spikes. Both the number of output spikes and the duration of the burst produced by the model directly reflect the temporal correlation of two input signals, and it is inversely proportional to the time difference. Milde et al. developed an analog Complementary Metal-Oxide-Semiconductor (CMOS) implementation of the TDE, characterized its performances on silicon and applied it to the encoding of Optical Flow (OF). The TDE model has already been used for processing visual (Schoepe et al., 2019), auditory and olfactory information. Its universal applicability has great potential for inspiring innovative pre-processing for SNNs, especially supporting closeloop neuromorphic systems with low latency requirements. This wide range of possible applications poses a challenge in terms of time resolution and scalability. Time resolution in analog circuits is constrained by the size of the capacitors. Therefore, for high time constants applications, large capacitors would be needed. Furthermore, mismatch problems and parameter setting difficulties may appear due to the analog nature of the implementation. In this section, a generic, event-based digital implementation of the Time Difference Encoder (TDE) model is presented. Its time resolution is configurable by means of a clock divider, covering a time range from nanoseconds to seconds (Gutierrez- Galan et al., 2021a). Moreover, the model can be deployed on FPGA-based platforms. This computational platform suits the integration of SNNs very well due to its 9A live demo video of this model running in real-time on a robotic platform can be found in https: //youtu.be/v56lpGJEkA4 222 Chapter 7. Bibliography Falanga, Davide, Suseong Kim, and Davide Scaramuzza (2019). “How fast is too fast? the role of perception latency in high-speed sense and avoid”. In: IEEE Robotics and Automation Letters 4.2, pp. 1884–1891. Falanga, Davide, Kevin Kleber, Stefano Mintchev, Dario Floreano, and Davide Scaramuzza (2018). “The foldable drone: A morphing quadrotor that can squeeze and fly”. In: IEEE Robotics and Automation Letters 4.2, pp. 209–216. Fang, Ying, Zhaofei Yu, and Feng Chen (2020). “Noise Helps Optimization Escape From Saddle Points in the Synaptic Plasticity”. In: Frontiers in neuroscience 14, p. 343. Finger, H., S. Liu, P. Ruvolo, and J. R. Movellan (2010). “Approaches and databases for online calibration of binaural sound localization for robotic heads”. In: 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4340–4345. DOI:10.1109/IROS.2010.5650515. Finger, Holger et al. (2011). “Estimating the location of a sound source with a spike-timing localization algorithm”. In: 2011 IEEE International Symposium of Circuits and Systems (ISCAS). IEEE, pp. 2461–2464. Fischer, Tim, Marco Caversaccio, and Wilhelm Wimmer (2020). “A front-back confusion metric in horizontal sound localization: The fbc score”. In: ACM Symposium on Applied Perception 2020, pp. 1–5. Fragnière, Eric (1998). Analogue VLSI emulation of the cochlea. Tech. rep. EPFL. Franken, Tom P, Philip X Joris, and Philip H Smith (2018). “Principal cells of the brainstem’s interaural sound level detector are temporal differentiators rather than integrators”. In: Elife 7, e33854. Frenkel, Charlotte et al. (2017). “A compact phenomenological digital neuron implementing the 20 Izhikevich behaviors”. In: 2017 IEEE Biomedical Circuits and Systems Conference (BioCAS). IEEE, pp. 1–4. Frenkel, Charlotte et al. (2019). “MorphIC: A 65-nm 738k-Synapse/mm Quad- Core Binary-Weight Digital Neuromorphic Processor with Stochastic Spike- Driven Online Learning”. In: arXiv preprint arXiv:1904.08513. Fries, Pascal (2005). “A mechanism for cognitive dynamics: neuronal communication through neuronal coherence”. In: Trends in cognitive sciences 9.10, pp. 474–480. Fujii, Hiroshi, Hiroyuki Ito, Kazuyuki Aihara, Natsuhiro Ichinose, and Minoru Tsukada (1996). “Dynamical cell assembly hypothesis—theoretical possibility of spatio-temporal coding in the cortex”. In: Neural Networks 9.8, pp. 1303–1350. Furber, S. B. et al. (2013a). “Overview of the SpiNNaker System Architecture”. In: IEEE Transactions on Computers 62.12, pp. 2454–2467. ISSN: 0018-9340. DOI: 10.1109/TC.2012.142. Furber, Stephen and Andrew Brown (2009). “Biologically-inspired massivelyparallel architectures-computing beyond a million processors”. In: Application of Concurrency to System Design, 2009. ACSD’09. Ninth International Conference On. IEEE, pp. 3–12. Chapter 7. Bibliography 223 Furber, Steve (2016). “Large-scale neuromorphic computing systems”. In: Journal of neural engineering 13.5, p. 051001. Furber, Steve B., Francesco Galluppi, Steve Temple, and Luis A. Plana (2014b). “The SpiNNaker Project”. In: Proceedings of the IEEE.ISSN: 00189219. DOI: 10.1109/JPROC.2014.2304638. Furber, Steve B, Francesco Galluppi, Steve Temple, and Luis A Plana (2014a). “The SpiNNaker project”. In: Proceedings of the IEEE 102.5, pp. 652–665. Furber, Steve B, David R Lester, Luis A Plana, Jim D Garside, Eustace Painkras, Steve Temple, and Andrew D Brown (2013b). “Overview of the SpiNNaker system architecture”. In: IEEE Transactions on Computers 62.12, pp. 2454–2467. Gambin, Isabel, Ivan Grech, Owen Casha, Edward Gatt, and Joseph Micallef (2010). “Digital cochlea model implementation using Xilinx XC3S500E spartan-3E FPGA”. In: Electronics, Circuits, and Systems (ICECS), 2010 17th IEEE International Conference on. IEEE, pp. 946–949. Gardner, Mark B (1968). “Proximity image effect in sound localization”. In: The Journal of the Acoustical Society of America 43.1, pp. 163–163. George, Suma, Sihwan Kim, Sahil Shah, Jennifer Hasler, Michelle Collins, Farhan Adil, Richard Wunderlich, Stephen Nease, and Shubha Ramakrishnan (2016). “A programmable and configurable mixed-mode FPAA SoC”. In: IEEE Transactions on Very Large Scale Integration (VLSI) Systems 24.6, pp. 2253–2261. Gerstner, Wulfram et al. (2002). Spiking neuron models: Single neurons, populations, plasticity. Cambridge university press. Glackin, Brendan, Julie A Wall, Thomas M McGinnity, Liam P Maguire, and Liam J McDaid (2010). “A spiking neural network model of the medial superior olive using spike timing dependent plasticity for sound localization”. In: Frontiers in computational neuroscience 4, p. 18. Glover, Arren et al. (2018). “The event-driven software library for YARP—With algorithms and iCub applications”. In: Frontiers in Robotics and AI 4, p. 73. Gómez-Rodríguez, F, A Jiménez-Fernández, F Pérez-Peña, L Miró, MJ Domínguez-Morales, A Ríos-Navarro, E Cerezuela, D Cascado-Caballero, and A Linares-Barranco (2016). “ED-Scorbot: A robotic test-bed framework for FPGA-based neuromorphic systems”. In: Biomedical Robotics and Biomechatronics (BioRob), 2016 6th IEEE International Conference on. IEEE, pp. 237–242. Gomez-Rodriguez, F. et al. (2005). “Two Hardware Implementations of the Exhaustive Synthetic AER Generation Method”. In: Computational Intelligence and Bioinspired Systems. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 534–540. ISBN: 978-3-540-32106-4. Gomez-Rodriguez, Francisco, Alejandro Linares-Barranco, L Miro, Shih-Chii Liu, André Van Schaik, Ralph Etienne-Cummings, and M Anthony Lewis (2007). “AER auditory filtering and CPG for robot control”. In: 2007 IEEE International Symposium on Circuits and Systems. IEEE, pp. 1201–1204. 224 Chapter 7. Bibliography Gonzalez-Billandon, Jonas, Alessandra Sciutti, Matthew Tata, Giulio Sandini, and Francesco Rea (2020). “Audiovisual cognitive architecture for autonomous learning of face localisation by a Humanoid Robot”. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 5979–5985. Grasse, Lukas, Sylvain J Boutros, and Matthew S Tata (2021). “Speech interaction to control a hands-free delivery robot for high-risk health care scenarios”. In: Frontiers in Robotics and AI 8. Grillner, Sten, Peter Wallén, Kazuya Saitoh, Alexander Kozlov, and Brita Robertson (2008). “Neural bases of goal-directed locomotion in vertebrates—an overview”. In: Brain research reviews 57.1, pp. 2–12. Grothe, Benedikt, Michael Pecka, and David McAlpine (2010). “Mechanisms of sound localization in mammals”. In: Physiological reviews 90.3, pp. 983–1012. Guertin, Pierre A. (2009). “The mammalian central pattern generator for locomotion”. In: Brain Research Reviews 62.1, pp. 45–56. ISSN: 0165-0173. DOI:10 . 1016 / J . BRAINRESREV . 2009 . 08 . 002.URL:https : / / www . sciencedirect.com/science/article/pii/S0165017309000812. Gutierrez-Galan, Daniel, Chiara Bartolozzi, Juan Pedro Dominguez-Morales, Angel Jimenez-Fernandez, and Alejandro Linares-Barranco (2022). “Towards the Neuromorphic Implementation of the Auditory Perception in the ICub Robotic Platform”. In: Neuro-Inspired Computational Elements Conference. NICE 2022. Association for Computing Machinery, 11–12. ISBN: 9781450395595. DOI:10 . 1145 / 3517343 . 3517347.URL:https : //doi.org/10.1145/3517343.3517347. Gutierrez-Galan, Daniel, Juan P Dominguez-Morales, Fernando Perez-Pena, Angel Jimenez-Fernandez, and Alejandro Linares-Barranco (2019a). “Live demonstration: neuromorphic robotics, from audio to locomotion through spiking CPG on SpiNNaker”. In: 2019 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, pp. 1–1. Gutierrez-Galan, Daniel, Juan P Dominguez-Morales, Fernando Perez-Peña, Angel Jimenez-Fernandez, and Alejandro Linares-Barranco (2020). “NeuroPod: a real-time neuromorphic spiking CPG applied to robotics”. In: Neurocomputing 381, pp. 10–19. Gutierrez-Galan, Daniel, Juan Pedro Dominguez-Morales, Angel Jimenez- Fernandez, Ricardo Tapiador-Morales, Antonio Rios-Navarro, and Alejandro Linares-Barranco (2019b). “A neuromorphic approach of the sound source localization task in real-time embedded systems: work-in- progress”. In: Proceedings of the International Conference on Embedded Software Companion, pp. 1–2. Gutierrez-Galan, Daniel, Thorben Schoepe, Juan P Dominguez-Morales, Angel Jimenez-Fernandez, Elisabetta Chicca, and Alejandro Linares-Barranco (2021a). “An event-based digital time difference encoder model implementation for neuromorphic systems”. In: IEEE Transactions on Neural Networks and Learning Systems. Chapter 7. Bibliography 225 Gutierrez-Galan, Daniel et al. (2021b). “OpenNAS: Open Source Neuromorphic Auditory Sensor HDL code generator for FPGA implementations”. In: Neurocomputing 436, pp. 35–38. Habets, Emanuel AP (2006). “Room impulse response generator”. In: Technische Universiteit Eindhoven, Tech. Rep 2.2.4, p. 1. Haessig, Germain et al. (2020). “Event-based computation for touch localization based on precise spike timing”. In: Frontiers in Neuroscience 14, p. 420. Hambrook, Dillon A et al. (2017). “A Bayesian computational basis for auditory selective attention using head rotation and the interaural time-difference cue”. In: PloS one 12.10. Hamilton, Tara Julia, Craig Jin, Andre Van Schaik, and Jonathan Tapson (2008). “An active 2-D silicon cochlea”. In: IEEE Transactions on biomedical circuits and systems 2.1, pp. 30–43. Hassentstein, B and W Reichardt (Jan. 1956). “Systemtheoretische Analyse der Zeit-, Reihenfolgen- und Vorzeichenauswertung bei der Bewegungsperzeption des Rüsselkäfers Chlorophanus”. In: Z. Naturforsch. 11b, pp. 513–524. DOI:10.1515/znb-1956-9-1004. Hodgkin, Alan L and Andrew F Huxley (1939). “Action potentials recorded from inside a nerve fibre”. In: Nature 144.3651, p. 710. Horiuchi, T. K. (2009). “A Spike-Latency Model for Sonar-Based Navigation in Obstacle Fields”. In: IEEE Transactions on Circuits and Systems I: Regular Papers 56.11, pp. 2393–2401. ISSN: 1549-8328. DOI:10.1109/TCSI.2009.2015597. Hull, Kerry L (2011). Human Form, Human Function: Essentials of Anatomy & Physiology. Lippincott Williams & Wilkins. Hynna, Kai and Kwabena Boahen (2001). “Space-rate coding in an adaptive silicon neuron”. In: Neural Networks 14.6-7, pp. 645–656. Iacono, Massimiliano, Stefan Weber, Arren Glover, and Chiara Bartolozzi (2018). “Towards event-driven object detection with off-the-shelf deep learning”. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, pp. 1–9. Iakymchuk, Taras, Alfredo Rosado, Teresa Serrano-Gotarredona, Bernabé Linares-Barranco, Angel Jiménez-Fernandez, Alejandro Linares-Barranco, and Gabriel Jiménez-Moreno (2014). “An AER handshake-less modular infrastructure PCB with x8 2.5 Gbps LVDS serial links”. In: 2014 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, pp. 1556–1559. Indiveri, G. et al. (2011a). “Neuromorphic silicon neuron circuits”. In: Frontiers in Neuroscience 5, pp. 1–23. ISSN: 1662-453X. DOI:10.3389/fnins.2011.00073. Indiveri, Giacomo, Elisabetta Chicca, and Rodney Douglas (2006). “A VLSI array of low-power spiking neurons and bistable synapses with spiketiming dependent plasticity”. In: IEEE transactions on neural networks 17.1, pp. 211–221. Indiveri, Giacomo et al. (2011b). “Neuromorphic silicon neuron circuits”. In: Frontiers in neuroscience 5, p. 73. Ito, Akira (1989). Barrel shifter. US Patent 4,829,460. 226 Chapter 7. Bibliography Jäckel, David, Rico Moeckel, and Shih-Chii Liu (2010). “Sound recognition with spiking silicon cochlea and Hidden Markov Models”. In: Ph. D. Research in Microelectronics and Electronics (PRIME), 2010 Conference on. IEEE, pp. 1–4. James, Robert (2020). “Spikes from sound: A model of the human auditory periphery on SpiNNaker”. In: Janus, Scott (2004). Audio in the 21st Century. Intel Press. Jeffress, Lloyd A (1948). “A place theory of sound localization.” In: Journal of comparative and physiological psychology 41.1, p. 35. Jimenez-Fernandez, Angel, Gabriel Jimenez-Moreno, Alejandro Linares- Barranco, Manuel J Dominguez-Morales, Rafael Paz-Vicente, and Anton Civit-Balcells (2012). “A neuro-inspired spike-based PID motor controller for multi-motor robots with low cost FPGAs”. In: Sensors 12.4, pp. 3831–3856. Jimenez-Fernandez, Angel et al. (2010). “Building blocks for spikes signals processing”. In: Neural Networks (IJCNN), The 2010 International Joint Conference on. IEEE, pp. 1–8. Jimenez-Fernandez, Angel et al. (2017). “A Binaural Neuromorphic Auditory Sensor for FPGA: A Spike Signal Processing Approach.” In: IEEE Trans. Neural Netw. Learning Syst. 28.4, pp. 804–818. Jiménez Fernández, Ángel Francisco (2010). “Diseño y evaluación de sistemas de control y procesamiento de señales basados en modelos neuronales pulsantes”. In: Johnston, Daniel and Samuel Miao-Sin Wu (1994). Foundations of cellular neurophysiology. MIT press. Jones, Simon, Ray Meddis, Seow Chuan Lim, and A Robert Temple (2000). “Toward a digital neuromorphic pitch extraction system”. In: IEEE transactions on neural networks 11.4, pp. 978–987. Kaas, John H. (1997). “Topographic Maps are Fundamental to Sensory Processing”. In: Brain Research Bulletin 44.2, pp. 107–112. Kandel, Eric R, James H Schwartz, Thomas M Jessell, Steven Siegelbaum, A James Hudspeth, and Sarah Mack (2000). Principles of neural science. Vol. 4. McGraw-hill New York. Kandler, Karl, Amanda Clause, and Jihyun Noh (2009). “Tonotopic reorganization of developing auditory brainstem circuits”. In: Nature neuroscience 12.6, pp. 711–717. Katsiamis, Andreas G, Emmanuel M Drakakis, and Richard F Lyon (2007). “Practical gammatone-like filters for auditory processing”. In: EURASIP Journal on Audio, Speech, and Music Processing 2007.1, p. 063685. Khacef, Lyes, Laurent Rodriguez, and Benoît Miramond (2020). “Brain-inspired self-organization with cellular neuromorphic computing for multimodal unsupervised learning”. In: Electronics 9.10, p. 1605. Kish, Lazlo B. (2002). “End of Moore’s law: thermal (noise) death of integration in microand nano electronics”. In: Physics Letters A 305, pp. 144–149. Chapter 7. Bibliography 227 Knight, James and Thomas Nowotny (2019). “GeNN: GPU-enhanced neural networks”. In: NEST Conference 2019. Knight, James C and Thomas Nowotny (2018). “GPUs outperform current HPC and neuromorphic solutions in terms of speed and energy when simulating a highly-connected cortical model”. In: Frontiers in neuroscience 12, p. 941. Knight, James C and Thomas Nowotny (2021). “Larger GPU-accelerated brain simulations with procedural connectivity”. In: Nature Computational Science 1.2, pp. 136–142. Kumar, Nagendra, Wolfgang Himmelbauer, Gert Cauwenberghs, and Andreas G Andreou (1998). “An analog VLSI chip with asynchronous interface for auditory feature extraction”. In: IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing 45.5, pp. 600–606. Lande, Tor Sverre, J-T Marienborg, and Yngvar Berg (2000). “Neuromorphic cochlea implants”. In: 2000 IEEE International Symposium on Circuits and Systems (ISCAS). Vol. 4. IEEE, pp. 401–404. Lazzaro, John and Carver Mead (1989a). “Circuit models of sensory transduction in the cochlea”. In: Analog VLSI Implementation of Neural Systems. Springer, pp. 85–101. Lazzaro, John and Carver A Mead (1989b). “A silicon model of auditory localization”. In: Neural computation 1.1, pp. 47–57. Lazzaro, John and John Wawrzynek (1995). “A multi-sender asynchronous extension to the AER protocol”. In: Advanced Research in VLSI, 1995. Proceedings., Sixteenth Conference on. IEEE, pp. 158–169. Lazzaro, John, John Wawrzynek, Misha Mahowald, Massimo Sivilotti, and Dave Gillespie (1993). “Silicon auditory processors as computer peripherals”. In: Advances in Neural Information Processing Systems, pp. 820–827. Lele, Ashwin, Yan Fang, Justin Ting, and Arijit Raychowdhury (2021). “An End- to-end Spiking Neural Network Platform for Edge Robotics: From Event- Cameras to Central Pattern Generation”. In: IEEE Transactions on Cognitive and Developmental Systems. Leong, Monk-Ping, Craig T Jin, and Philip HW Leong (2003). “An FPGA-based electronic cochlea”. In: EURASIP Journal on Applied Signal Processing 2003, pp. 629–638. Levi, Timothée et al. (2018). “Digital implementation of Hodgkin–Huxley neuron model for neurological diseases studies”. In: Artificial Life and Robotics. Lichtsteiner, Patrick et al. (2008). “A 128×128 120 dB 15µs Latency Asynchronous Temporal Contrast Vision Sensor”. In: IEEE journal of solid-state circuits 43.2, pp. 566–576. Linares-Barranco, A. et al. (2006). “On algorithmic rate-coded AER generation”. In: IEEE Transactions on Neural Networks 17.3, pp. 771–788. Linares-Barranco, Alejandro, Fernando Perez-Peña, Angel Jimenez-Fernandez, and Elisabetta Chicca (2020). “ED-BioRob: a Neuromorphic Robotic Arm with FPGA-based infrastructure for Bio-inspired spiking motor controllers”. In: Frontiers in Neurorobotics 14. 228 Chapter 7. Bibliography Liu, Jindong, Harry Erwin, and Stefan Wermter (2008). “Mobile robot broadband sound localisation using a biologically inspired spiking neural network”. In: 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, pp. 2191–2196. Liu, Jindong, David Perez-Gonzalez, Adrian Rees, Harry Erwin, and Stefan Wermter (2010a). “A biologically inspired spiking neural network model of the auditory midbrain for sound source localisation”. In: Neurocomputing 74.1-3, pp. 129–139. Liu, Qian, Cameron Patterson, Steve Furber, Zhangqin Huang, Yibin Hou, and Huibing Zhang (2013). “Modeling populations of spiking neurons for fine timing sound localization”. In: The 2013 International Joint Conference on Neural Networks (IJCNN). IEEE, pp. 1–8. Liu, Shih-Chii, Tobi Delbruck, Giacomo Indiveri, Adrian Whatley, and Rodney Douglas (2015). Event-based neuromorphic systems. John Wiley & Sons. Liu, Shih-Chii, Jörg Kramer, Giacomo Indiveri, Tobias Delbrück, and Rodney Douglas (2002). Analog VLSI: circuits and principles. MIT press. Liu, Shih-Chii, André van Schaik, Bradley A Minch, and Tobi Delbruck (2014). “Asynchronous Binaural Spatial Audition Sensor With 2x64x4 Channel Output”. In: IEEE transactions on biomedical circuits and systems 8.4, pp. 453–464. Liu, Shih-Chii, André Van Schaik, Bradley A Mincti, and Tobi Delbruck (2010b). “Event-based 64-channel binaural silicon cochlea with Q enhancement mechanisms”. In: Circuits and Systems (ISCAS), Proceedings of 2010 IEEE International Symposium on. IEEE, pp. 2027–2030. Liu, Weimin, Andreas G Andreou, and MH Goldstein (1991). “An analog integrated speech front-end based on the auditory periphery”. In: Neural Networks, 1991., IJCNN-91-Seattle International Joint Conference on. Vol. 2. IEEE, pp. 861–864. Liu, Weimin, Andreas G Andreou, and Moise H Goldstein (1992). “Voiced-speech representation by an analog silicon model of the auditory periphery”. In: IEEE Transactions on Neural Networks 3.3, pp. 477–487. Liu, Ying, Khaled Benkrid, AbdSamad Benkrid, and Server Kasap (2009). “An fpga-based web server for high performance biological sequence alignment”. In: 2009 NASA/ESA Conference on Adaptive Hardware and Systems. IEEE, pp. 361–368. Lopez-Poveda, Enrique A (2018). “Olivocochlear efferents in animals and humans: from anatomy to clinical relevance”. In: Frontiers in neurology 9, p. 197. Lyon, Richard (1982). “A computational model of filtering, detection, and compression in the cochlea”. In: Acoustics, Speech, and Signal Processing, IEEE International Conference on ICASSP’82. Vol. 7. IEEE, pp. 1282–1285. Lyon, Richard F (2017). Human and machine hearing. Cambridge University Press. Lyon, Richard F and Carver Mead (1988). “An analog electronic cochlea”. In: IEEE Transactions on Acoustics, Speech, and Signal Processing 36.7, pp. 1119–1134. Chapter 7. Bibliography 229 Maass, Wolfgang and Christopher M Bishop (2001). Pulsed neural networks. MIT press. Mahowald, Misha (1992). “VLSI analogs of neuronal visual processing: a synthesis of form and function”. Maqueda, Ana I, Antonio Loquercio, Guillermo Gallego, Narciso García, and Davide Scaramuzza (2018). “Event-based vision meets deep learning on steering prediction for self-driving cars”. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5419–5427. Massaro, Dominic W and David G Stork (1998). “Speech recognition and sensory integration: a 240-year-old theorem helps explain how people and machines can integrate auditory and visual information to understand speech”. In: American Scientist 86.3, pp. 236–244. Massoud, Tarek M and Timothy K Horiuchi (2010). “A neuromorphic VLSI head direction cell system”. In: IEEE Transactions on Circuits and Systems I: Regular Papers 58.1, pp. 150–163. McDonald, John J, Wolfgang A Teder-SaÈlejaÈrvi, and Steven A Hillyard (2000). “Involuntary orienting to sound improves visual perception”. In: Nature 407.6806, pp. 906–908. Mead, Carver (1989). Analog VLSI and Neural Systems. Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc. ISBN: 0-201-05992-4. Mead, Carver (1990). “Neuromorphic electronic systems”. In: Proceedings of the IEEE 78.10, pp. 1629–1636. Metta, Giorgio, Paul Fitzpatrick, and Lorenzo Natale (2006). “YARP: yet another robot platform”. In: International Journal of Advanced Robotic Systems 3.1, p. 8. Milde, Moritz B, Olivier JN Bertrand, Ryad Benosman, Martin Egelhaaf, and Elisabetta Chicca (2015). “Bioinspired event-driven collision avoidance algorithm based on optic flow”. In: Event-based Control, Communication, and Signal Processing (EBCCSP), 2015 International Conference on. IEEE, pp. 1–7. Milde, Moritz B, Hermann Blum, Alexander Dietmüller, Dora Sumislawska, Jörg Conradt, Giacomo Indiveri, and Yulia Sandamirskaya (2017). “Obstacle avoidance and target acquisition for robot navigation using a mixed signal analog/digital neuromorphic processing system”. In: Frontiers in neurorobotics 11, p. 28. Milde, Moritz B et al. (2018). “Spiking elementary motion detector in neuromorphic systems”. In: Neural computation 30.9, pp. 2384–2417. Minassian, Karen, Ursula S Hofstoetter, Florin Dzeladini, Pierre A Guertin, and Auke Ijspeert (2017). “The Human Central Pattern Generator for Locomotion: Does It Exist and Contribute to Walking?” In: The Neuroscientist 23.6, pp. 649–663. Miró Amarante, María Lourdes (2013). “Una aportación al procesado neuromórfico de audio basado en modelos pulsantes. Desde la cóclea a la percepción auditiva”. Molholm, Sophie and John J Foxe (2005). “Look ‘hear’, primary auditory cortex is active during lip-reading”. In: Neuroreport 16.2, pp. 123–124. 230 Chapter 7. Bibliography Monforte, Marco, Ander Arriandiaga, Arren Glover, and Chiara Bartolozzi (2020). “Where and When: Event-Based Spatiotemporal Trajectory Prediction from the iCub’s Point-Of-View”. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 9521–9527. Montealegre, Norma, David Merodio, Agustin Fernandez, and Philippe Armbruster (2015). “In-flight reconfigurable FPGA-based space systems”. In: 2015 NASA/ESA Conference on Adaptive Hardware and Systems (AHS). IEEE, pp. 1–8. Moradi, Saber, Ning Qiao, Fabio Stefanini, and Giacomo Indiveri (2017). “A scalable multicore architecture with heterogeneous memory structures for dynamic neuromorphic asynchronous processors (DYNAPs)”. In: IEEE transactions on biomedical circuits and systems 12.1, pp. 106–122. Moravec, Hans (1998). “When will computer hardware match the human brain?” In: Jouranl of Evolution and Technology 1. Mugliette, Christian, Ivan Grech, Owen Casha, Edward Gatt, and Joseph Micallef (2011). “FPGA active digital cochlea model”. In: Electronics, Circuits and Systems (ICECS), 2011 18th IEEE International Conference on. IEEE, pp. 699–702. Müller, G. R. and J. Conradt (2011). “A miniature low-power sensor system for real time 2D visual tracking of LED markers”. In: 2011 IEEE International Conference on Robotics and Biomimetics, pp. 2429–2434. DOI:10.1109/ROBIO. 2011.6181669. Natale, Lorenzo, Chiara Bartolozzi, Francesco Nori, Giulio Sandini, and Giorgio Metta (2021). “The iCub platform: evolution and current trends”. In: arXiv e-prints, arXiv–2105. Natale, Lorenzo, Chiara Bartolozzi, Daniele Pucci, Agnieszka Wykowska, and Giorgio Metta (2017). “icub: The not-yet-finished story of building a robot child”. In: Science Robotics 2.13. Nathan, Adam (2006). Windows presentation foundation unleashed. Pearson Education. Naveros, Francisco, Niceto R Luque, Eduardo Ros, and Angelo Arleo (2019). “VOR adaptation on a humanoid iCub robot using a spiking cerebellar model”. In: IEEE transactions on cybernetics 50.11, pp. 4744–4757. Oster, Matthias, Rodney Douglas, and Shih-Chii Liu (July 2009). “Computation with Spikes in a Winner-Take-All Network”. In: Neural computation 21, pp. 2437–65. DOI:10.1162/neco.2009.07-08-829. Painkras, Eustace et al. (2013). “SpiNNaker: A 1-W 18-core system-on-chip for massively-parallel neural network simulation”. In: IEEE Journal of Solid-State Circuits 48.8, pp. 1943–1953. Park, Paul KJ et al. (2013). “Fast neuromorphic sound localization for binaural hearing aids”. In: 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE. Parmiggiani, Alberto, Marco Maggiali, Lorenzo Natale, Francesco Nori, Alexander Schmitz, Nikos Tsagarakis, Jose Santos Victor, Francesco Becchi, Chapter 7. Bibliography 231 Giulio Sandini, and Giorgio Metta (2012). “The design of the iCub humanoid robot”. In: International journal of humanoid robotics 9.04, p. 1250027. Pasquale, Giulia, Carlo Ciliberto, Francesca Odone, Lorenzo Rosasco, and Lorenzo Natale (2015). “Teaching iCub to recognize objects using deep Convolutional Neural Networks”. In: Machine Learning for Interactive Systems. PMLR, pp. 21–25. Patton, Kevin T, Gary A Thibodeau, and Matthew M Douglas (2012). Essentials of anatomy & physiology. Elsevier/Mosby. Pavlou, Athanasios and Matthew Casey (2010). “Simulating the effects of cortical feedback in the superior colliculus with topographic maps”. In: The 2010 International Joint Conference on Neural Networks (IJCNN). IEEE, pp. 1–8. Perez-Peña, Fernando, Arturo Morgado-Estevez, Alejandro Linares-Barranco, Angel Jimenez-Fernandez, Francisco Gomez-Rodriguez, Gabriel Jimenez- Moreno, and Juan Lopez-Coronado (2013a). “Neuro-inspired spike-based motion: from dynamic vision sensor to robot motor open-loop control through spike-VITE”. In: Sensors 13.11, pp. 15805–15832. Perez-Peña, Fernando, Arturo Morgado-Estevez, Alejandro Linares-Barranco, Angel Jimenez-Fernandez, Francisco Gomez-Rodriguez, Gabriel Jimenez- Moreno, and Juan Lopez-Coronado (2013b). “Neuro-Inspired Spike-Based Motion: From Dynamic Vision Sensor to Robot Motor Open-Loop Control through Spike-VITE”. In: Sensors 13.11, 15805–15832. ISSN: 1424-8220. Perez-Peña, Fernando et al. (2019). “Digital neuromorphic real-time platform”. In: Neurocomputing. Plana, L.A., J. Heathcote, J.S. Pepper, S. Davidson, J. Garside, S. Temple, and S.B. Furber (2014). “spI/O: A library of FPGA designs and reusable modules for I/O in SpiNNaker systems”. In: DOI:10.5281/zenodo.51476. Plana, Luis A, Steve B Furber, Steve Temple, Mukaram Khan, Yebin Shi, Jian Wu, and Shufan Yang (2007). “A GALS infrastructure for a massively parallel multiprocessor”. In: IEEE Design & Test of Computers 24.5. Plenge, Georg (1974). “On the differences between localization and lateralization”. In: The Journal of the Acoustical Society of America 56.3, pp. 944–951. Posch, Christoph, Daniel Matolin, and Rainer Wohlgenannt (2010). “A QVGA 143 dB dynamic range frame-free PWM image sensor with lossless pixel-level video compression and time-domain CDS”. In: IEEE Journal of Solid-State Circuits 46.1, pp. 259–275. Qiao, Ning, Hesham Mostafa, Federico Corradi, Marc Osswald, Fabio Stefanini, Dora Sumislawska, and Giacomo Indiveri (2015). “A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128K synapses”. In: Frontiers in neuroscience 9, p. 141. Rakic, Pasko (1988). “Specification of cerebral cortical areas”. In: Science 241.4862, pp. 170–176. Rasetto, Marco, Juan P Dominguez-Morales, Angel Jimenez-Fernandez, and Ryad Benosman (2021). “Event Based Time-Vectors for auditory features 239 Appendix A OpenNAS software tool A.1 OpenNAS screens summary 1. Software metadata: • Current software version: v1.1.34. • Permanent link to executables of this version: https://github.com/ RTC-research-group/OpenNAS/releases/tag/1.1.34. • Permanent link to code/repository used of this code version: http: //github.com/RTC-research-group/OpenNAS. • Software License: GNU General Public License (GPL). • Computing platform/Operating System: Microsoft Windows. • Installation requirements & dependencies: Microsoft .NET Framework 4.5 or greater. • Software code languages, tools, and services used: C# and VHDL. • Link to user manual: http://github.com/RTC-research-group/ OpenNAS/tree/master/OpenNAS/Wiki_files/User_manual/. • Link to developer documentation or manual: https://github. com/RTC-research-group/OpenNAS/tree/master/OpenNAS/ SandCastleBuilder/Help. • Support email for questions: [email protected]. 240 Appendix A. OpenNAS software tool A.1.1 OpenNAS welcome screen FIGURE A.1: OpenNAS tool welcome message. A.1. OpenNAS screens summary 241 A.1.2 OpenNAS common settings screen FIGURE A.2: OpenNAS screen for step 1: common settings. 242 Appendix A. OpenNAS software tool A.1.3 OpenNAS input interface screen FIGURE A.3: OpenNAS screen for step 2: audio input interface. A.1. OpenNAS screens summary 243 A.1.4 OpenNAS processing architecture screen FIGURE A.4: OpenNAS screen for step 3: audio processing architecture. 244 Appendix A. OpenNAS software tool A.1.5 OpenNAS output interface screen FIGURE A.5: OpenNAS screen for step 4: spiking output interface. A.1. OpenNAS screens summary 245 A.1.6 OpenNAS destination folder screen FIGURE A.6: OpenNAS screen for step 5: destination folder selection. 246 Appendix A. OpenNAS software tool A.1.7 OpenNAS generation success screen FIGURE A.7: OpenNAS final screen: code generator statistics. 247 Appendix B NASIC test PCB B.1 NASIC test PCB files 1. Features: • 2 PDM microphones interface. • 1 CS5343 audio ADC with I2S interface. • 1 3.5mm audio jack female connector. • 1 generic purpose jumper. • 10-pin connector for FPGA interface. • 3.3V power supply. 2. Hardware requirements: • Any FPGA- or microcontoller-based board with I2S interface. 254 Appendix C. Generic purpose PCBs C.1.1 Schematic: ADC_PDM_mic_board.SchDoc 1 1 2 2 3 3 4 4 D D C C B B A A Title Number RevisionSize A4 Date: 03/05/2021 Sheet of File: D:\Proyectos\..\ADC_PDMNodev2.SchDocDrawn By: PDM_CLK 1 GND 2 PDM_DAT 3 NC 4 L/R 5 VCC 6 UL Component_1 PDM_CLK 1 GND 2 PDM_DAT 3 NC 4 L/R 5 VCC 6 UR Component_1 PDM_CLK_L PDM_CLK_R PDM_DAT_L PDM_DAT_R VCC_3.3 VCC_3.3 VCC_3.3 VCC_3.3 GND GND 1 2 3 4 5 6 7 8 910 P0 Header 5X2 GND VCC_3.3 PDM_CLK_L PDM_DAT_L PDM_CLK_R PDM_DAT_R SDOUT SCLK LRCK 4 2 3 1 J0 Phonejack3 RN GND 5.1K R0 Res1 5.1K R1 Res1 5.1K R2 Res1 5.1K R3 Res1 GND GND 1uF C0 180pF C1 Cap GND 1uF C2 Cap 180pF C3 Cap GND CS5343.AINR CS5343.AINL Passive Analog Input 1 2 P1 Header 2 SSEL 100uF C4 T495C VCC_3.3 0.1uF C5 Cap VCC_VA GND VA select SDOUT 1 SCLK 2 LRCK 3 MCLK 4 FILT+ 5AINL 6 VQ 7 AINR 8 GND 9 VA 10 ADC CS5343/4 OE/_ST 1 GND 2 OUT 3 VCC 4 Y1 SG7050CAN/CBN/CCN 10K R7 Res1 0RR5 0RR6 0RR4 10K R8 Res1 VCC_VA VCC_VA SDOUT SCLK LRCK MCLK 0.1uF C6 Cap 1uF C7 Cap GND 0.1uF C8 Cap 1uF C9 Cap GND CS5343.AINL CS5343.AINR 0.1uF C10 Cap 1uF C11 Cap GND GND 0RR9 VCC_VA 1000pF C12 0.1uF C13 Cap GND 10R R10 Res1 No Pop R11 Res1 VCC_3.3 MCLK GND 0.1uF C14 Cap 1000pF C15 GND Crystal OscillatorClock freq: 24.576 MHz GND SSEL SDOUT: OUT SCLK: IN/OUT LRCK: IN/OUT MCLK: IN SCLK-LRCK: Optional pull-up resistor for configuring clocks in Master Mode as described in the "Master Mode Speed Selection" section on page 13 Pull-up to VA for Master Mode Pull-down to GND for Slave mode Place the capacitor close to the chip pin. And trace the line under the oscillator. PIADC01 PIADC02 PIADC03 PIADC04 PIADC05 PIADC06 PIADC07 PIADC08 PIADC09 PIADC010 COADC PIC001 PIC002 COC0 PIC101 PIC102 COC1 PIC201PIC202 COC2 PIC301 PIC302 COC3 PIC401 PIC402 COC4 PIC501 PIC502 COC5 PIC601 PIC602 COC6 PIC701 PIC702 COC7 PIC801 PIC802 COC8 PIC901 PIC902 COC9 PIC1001 PIC1002 COC10 PIC1101 PIC1102 COC11 PIC1201 PIC1202 COC12 PIC1301 PIC1302 COC13 PIC1401 PIC1402 COC14 PIC1501 PIC1502 COC15 PIJ001 PIJ002 PIJ003 PIJ004 COJ0 PIP001 PIP002 PIP003 PIP004 PIP005 PIP006 PIP007 PIP008 PIP009 PIP0010 COP0 PIP101 PIP102 COP1 PIR001 PIR002 COR0 PIR101 PIR102 COR1 PIR201PIR202 COR2 PIR301 PIR302 COR3 PIR401 PIR402 COR4 PIR501 PIR502 COR5 PIR601 PIR602 COR6 PIR701 PIR702 COR7 PIR801 PIR802 COR8 PIR901 PIR902 COR9 PIR1001 PIR1002 COR10 PIR1101 PIR1102 COR11 PIUL01 PIUL02 PIUL03 PIUL04 PIUL05 PIUL06 COUL PIUR01 PIUR02 PIUR03 PIUR04 PIUR05 PIUR06 COUR PIY101 PIY102 PIY103PIY104 COY1 PIADC06 PIC201 PIC301 NLCS53430AINL PIADC08 PIC002 PIC101 NLCS53430AINR PIADC09 PIC102 PIC302 PIC402 PIC502 PIC602PIC702 PIC802 PIC902 PIC1002 PIC1102 PIC1202PIC1302 PIC1401 PIC1501 PIJ002 PIP001 PIP101 PIR102 PIR302 PIUL02 PIUR02 PIY102 PIP008 PIR601 PIR802 NLLRCK PIADC04 PIY103 NLMCLK PIADC01 PIR402 PIR702 PIADC02 PIR502 PIADC03 PIR602 PIADC05 PIC601PIC701 PIADC07 PIC801 PIC901 PIADC010 PIC1001 PIC1101 PIR901 PIC001PIR002 PIR101 PIC202PIR201 PIR301 PIC1201PIC1301 PIC1402 PIC1502 PIR1002 PIR1101 PIY104 PIJ001 PIJ003 PIR001 PIJ004 PIR202 PIR1102 PIY101 PIUL04 PIUR04 PIP005 PIUL01 NLPDM0CLK0L PIP007 PIUR01 NLPDM0CLK0R PIP003 PIUL03 NLPDM0DAT0L PIP009 PIUR03 NLPDM0DAT0R PIP006 PIR501 NLSCLK PIP004 PIR401 NLSDOUT PIP0010 PIP102 NLSSEL PIC401 PIC501 PIP002 PIR701 PIR801 PIR902 PIR1001 PIUL05 PIUL06 PIUR05 PIUR06 FIGURE C.1: ADC-PDM microphones board PCB schematic. C.1. ADC-PDM microphones board PCB files 255 C.1.2 Board top: ADC_PDM_mic_board.PcbDoc PAADC06 PAADC07 PAADC08 PAADC09 PAADC010 PAADC05 PAADC04 PAADC03 PAADC02 PAADC01 COADC PAC002 PAC001 COC0 PAC102 PAC101 COC1 PAC202 PAC201 COC2 PAC302 PAC301 COC3 PAC401 PAC402 COC4 PAC502 PAC501 COC5 PAC602 PAC601 COC6 PAC702 PAC701 COC7 PAC802 PAC801 COC8 PAC902 PAC901 COC9 PAC1002 PAC1001 COC10 PAC1102 PAC1101 COC11 PAC1202 PAC1201 COC12 PAC1302 PAC1301 COC13 PAC1402 PAC1401 COC14 PAC1502 PAC1501 COC15 PALINEIN02 PALINEIN01 PALINEIN04 PALINEIN03 COLINEIN PAP0010 PAP009 PAP008 PAP007 PAP006 PAP005 PAP004 PAP003 PAP002 PAP001 COP0 PAR002 PAR001 COR0 PAR102 PAR101 COR1 PAR202 PAR201 COR2 PAR302 PAR301 COR3 PAR402 PAR401 COR4 PAR502 PAR501 COR5 PAR602 PAR601 COR6 PAR702 PAR701 COR7 PAR802 PAR801 COR8 PAR902 PAR901 COR9 PAR1002 PAR1001 COR10 PAR1102 PAR1101 COR11 PASSEL01 PASSEL02 COSSEL PAUL03 PAUL02 PAUL01 PAUL06 PAUL05 PAUL04 COUL PAUR03 PAUR02 PAUR01 PAUR06 PAUR05 PAUR04 COUR PAY104PAY103 PAY102 PAY101 COY1 PAADC06 PAC201 PAC301 PAADC08 PAC002 PAC101 PAADC09 PAC102 PAC302 PAC402 PAC502 PAC602 PAC702 PAC802 PAC902 PAC1002PAC1102 PAC1202PAC1302 PAC1401 PAC1501 PALINEIN02 PAP001 PAR102 PAR302 PASSEL01 PAUL02 PAUR02 PAY102 PAP008 PAR601PAR802 PAADC04 PAY103 PAADC01 PAR402PAR702 PAADC02 PAR502 PAADC03 PAR602 PAADC05 PAC601 PAC701 PAADC07 PAC801 PAC901 PAADC010 PAC1001PAC1101 PAR901 PAC001 PAR002 PAR101 PAC202 PAR201 PAR301 PAC1201PAC1301 PAC1402 PAC1502 PAR1002 PAR1101 PAY104 PALINEIN03 PAR001 PALINEIN04 PAR202 PAR1102 PAY101 PAP005 PAUL01 PAP007 PAUR01 PAP003 PAUL03 PAP009 PAUR03 PAP006 PAR501 PAP004 PAR401 PAP0010 PASSEL02 PAC401 PAC501 PAP002 PAR701 PAR801 PAR902 PAR1001 PAUL05 PAUL06 PAUR05 PAUR06 FIGURE C.2: ADC-PDM microphones board PCB board top view. FIGURE C.3: ADC-PDM microphones board PCB board top 3D view. C.1.3 Board bottom: ADC_PDM_mic_board.PcbDoc PAADC06 PAADC07 PAADC08 PAADC09 PAADC010 PAADC05 PAADC04 PAADC03 PAADC02 PAADC01 COADC PAC002 PAC001 COC0 PAC102 PAC101 COC1 PAC202 PAC201 COC2 PAC302 PAC301 COC3 PAC401 PAC402 COC4 PAC502 PAC501 COC5 PAC602 PAC601 COC6 PAC702 PAC701 COC7 PAC802 PAC801 COC8 PAC902 PAC901 COC9 PAC1002 PAC1001 COC10 PAC1102 PAC1101 COC11 PAC1202 PAC1201 COC12 PAC1302 PAC1301 COC13 PAC1402 PAC1401 COC14 PAC1502 PAC1501 COC15 PALINEIN02 PALINEIN01 PALINEIN04 PALINEIN03 COLINEIN PAP0010 PAP009 PAP008 PAP007 PAP006 PAP005 PAP004 PAP003 PAP002 PAP001 COP0 PAR002 PAR001 COR0 PAR102 PAR101 COR1 PAR202 PAR201 COR2 PAR302 PAR301 COR3 PAR402 PAR401 COR4 PAR502 PAR501 COR5 PAR602 PAR601 COR6 PAR702 PAR701 COR7 PAR802 PAR801 COR8 PAR902 PAR901 COR9 PAR1002 PAR1001 COR10 PAR1102 PAR1101 COR11 PASSEL01 PASSEL02 COSSEL PAUL03 PAUL02 PAUL01 PAUL06 PAUL05 PAUL04 COUL PAUR03 PAUR02 PAUR01 PAUR06 PAUR05 PAUR04 COUR PAY104PAY103 PAY102 PAY101 COY1 PAADC06 PAC201 PAC301 PAADC08 PAC002 PAC101 PAADC09 PAC102 PAC302 PAC402 PAC502 PAC602 PAC702 PAC802 PAC902 PAC1002PAC1102 PAC1202PAC1302 PAC1401 PAC1501 PALINEIN02 PAP001 PAR102 PAR302 PASSEL01 PAUL02 PAUR02 PAY102 PAP008 PAR601PAR802 PAADC04 PAY103 PAADC01 PAR402PAR702 PAADC02 PAR502 PAADC03 PAR602 PAADC05 PAC601 PAC701 PAADC07 PAC801 PAC901 PAADC010 PAC1001PAC1101 PAR901 PAC001 PAR002 PAR101 PAC202 PAR201 PAR301 PAC1201PAC1301 PAC1402 PAC1502 PAR1002 PAR1101 PAY104 PALINEIN03 PAR001 PALINEIN04 PAR202 PAR1102 PAY101 PAP005 PAUL01 PAP007 PAUR01 PAP003 PAUL03 PAP009 PAUR03 PAP006 PAR501 PAP004 PAR401 PAP0010 PASSEL02 PAC401 PAC501 PAP002 PAR701 PAR801 PAR902 PAR1001 PAUL05 PAUL06 PAUR05 PAUR06 FIGURE C.4: ADC-PDM microphones board PCB board bottom view. FIGURE C.5: ADC-PDM microphones board PCB board bottom 3D view. 256 Appendix C. Generic purpose PCBs C.2 ZTEX 2.13 base board PCB files 1. Features: • 8 general purpose LEDs. • 4 general purpose buttons. • 2 general purpose jumpers. • SpiNNaker interface through SpiNN-link connector. • ADC-PDM microphones board interface (see C.1). • CAVIAR-20 connector with 20 I/O pins. • JTAG connector for FPGA programming. 2. Hardware requirements: • ZTEX 2.13 board: from https://www.ztex.de/usb-fpga-2/ usb-fpga-2.13.e.html. C.2. ZTEX 2.13 base board PCB files 257 C.2.1 Schematic: ZTEX_base_board.SchDoc 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 D D C C B B A A Title Number RevisionSize A3 Date: 04/05/2021 Sheet of File: D:\Proyectos\..\NAS_ZTEX_Top.SchDoc Drawn By: 1 2 3 4 5 6 7 8 910 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 PAB1 Header 30X2 1 2 3 4 5 6 7 8 910 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 PCD1 Header 30X2 1 2 3 4 PAB2 Header 2X2 1 2 3 4 PCD2 Header 2X2 VIN VIN GND GND GNDGND GNDGND GND GNDGND GND GND USB+5 USB+5 VIN 1 2 PVSUP Header 2H 3.3V 3.3V 3.3V 3.3V 3.3V 3.3V 3.3V 3.3V IO_A3 IO_A4 IO_A5 IO_A6 IO_A7 IO_A8 IO_A9 IO_A10 IO_A11 IO_A12 IO_A13 IO_A14 IO_A18 IO_A19 IO_A20 IO_A21 IO_A22 IO_A23 IO_A24 IO_A25 IO_A26 IO_A27 IO_A28 IO_A29 IO_A30 IO_B3 IO_B4 IO_B5 IO_B6 IO_B7 IO_B8 IO_B9 IO_B10 IO_B11 IO_B12 IO_B13 IO_B14 IO_B18 IO_B19 IO_B20 IO_B21 IO_B22 IO_B23 IO_B24 IO_B25 IO_B26 IO_B27 IO_B28 IO_B29 IO_B30 IO_C3 IO_C4 IO_C5 IO_C6 IO_C7 IO_C8 IO_C9 IO_C10 IO_C11 IO_C12 IO_C13 IO_C14 IO_C15 IO_C19 IO_C20 IO_C21 IO_C22 IO_C23 IO_C24 IO_C25 IO_C26 IO_C27 IO_C28 IO_C29 IO_C30 IO_D3 IO_D4 IO_D5 IO_D6 IO_D7 IO_D8 IO_D9 IO_D10 IO_D11 IO_D12 IO_D13 IO_D14 IO_D15 IO_D19 IO_D20 IO_D21 IO_D22 IO_D23 IO_D24 IO_D25 IO_D26 IO_D27 IO_D28 IO_D29 IO_D30 JTAG_TDOJTAG_TDI JTAG_TCLK JTAG_TMS JTAG_VIO 1 2 3 4 5 6 7 8 910 11 12 13 14 P1 Xilinx Parallel IV ConnectorGND RESET JTAG_TDO JTAG_TDI JTAG_TCLK JTAG_TMS JTAG_VIO 1 2 3 4 5 6 7 8 910 P_GP1 Header 5X2 3.3V GP1_0 GP1_2 GP1_4 GP1_6 GND PB0 R26 4k7 R25 10k C20 0.1 uF 3.3V GND GND PB1 R3 4k7 R1 10k C21 0.1 uF 3.3V GND GND PB2 R28 4k7 R27 10k C22 0.1 uF GND GND PB3 R29 4k7 R2 10k C23 0.1 uF GND GND 3.3V 3.3V PRE_P PRE_N OUT_P OUT_N 330 RL0 Res Semi LED_0 GND 330 RL1 Res Semi LED_1 GND 330 RL2 Res Semi LED_2 GND 330 RL3 Res Semi LED_4 GND 330 RL4 Res Semi LED_5 GND 330 RL5 Res Semi LED_6 GND 330 RL6 Res Semi LED_7 GND 330 RL7 Res Semi LED_8 GND AER_ACKAER_REQ GND AER_0 AER_1 AER_2 AER_3 AER_4 AER_5 AER_6 AER_7 AER_8 1 2 3 4 5 6 7 8 910 11 12 13 14 15 16 17 18 19 20 JP12 AER SequencerOUT AER_9 AER_10 AER_11 AER_12 AER_13 AER_14 AER_15 GND PB4 RESET GND D0 D1 D2 D3 D4 D5 D6 D7 IO_A3 IO_A4 IO_D3 IO_D4 IO_C3 IO_C4 IO_B4 IO_B3 IO_D15 IO_D19IO_A18 IO_A19 IO_A8 IO_B8 IO_A9 IO_B9 IO_A10 IO_B10 IO_A11 IO_B11 IO_C20 IO_D20 IO_C21 IO_C22 IO_C23 IO_C24 IO_C25 IO_C26 IO_C27 IO_C28 IO_D21 IO_D22 IO_D23 IO_D24 IO_D25 IO_D26 IO_D27 IO_D28 PDM_CLK_L PDM_CLK_R PDM_DAT_L PDM_DAT_R +C38 10 uF +C39 10 uF GND 1 IN 3OUT 4 LM1117-1.8 LM1117 1.8VUSB+5 VCCA 1 A1 2 A2 3 A3 4 A4 5 A5 6 A6 7 A7 8 A8 9 GND 10 OE 11 B8 12 B7 13 B6 14 B5 15 B4 16 B3 17 B2 18 B1 19 VCCB 20 U1 NLSV8T244 VCCA 1 A1 2 A2 3 A3 4 A4 5 A5 6 A6 7 A7 8 A8 9 GND 10 OE 11 B8 12 B7 13 B6 14 B5 15 B4 16 B3 17 B2 18 B1 19 VCCB 20 U2 NLSV8T244 3.3V 3.3V 1.8V 1.8V AER_0 AER_1 AER_2 AER_3 AER_4 AER_5 AER_6 AER_7 AER_8 ACK_IN 1 2 3 4 5 6 7 8 910 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 P2 Header 17X2 ACK_IN DATA_IN_0 DATA_IN_1 DATA_IN_2 DATA_IN_3 DATA_IN_4 DATA_IN_5 DATA_IN_6 DATA_OUT_0 ACK_OUT DATA_OUT_1 DATA_OUT_2 DATA_OUT_3 DATA_OUT_4 DATA_OUT_5 DATA_OUT_6 DATA_IN_0 DATA_IN_1 DATA_IN_2 DATA_IN_3 DATA_IN_4 DATA_IN_5 DATA_IN_6 ACK_OUT DATA_OUT_0 DATA_OUT_1 DATA_OUT_2 DATA_OUT_3 DATA_OUT_4 DATA_OUT_5 DATA_OUT_6AER_9 AER_10 AER_11 AER_12 AER_13 AER_14 AER_15 GND GNDGND GNDGND GND GND 1 2 SPINN Header 2H 1 2 AEROUT Header 2H GND GND E_SPINN E_AEROUT IO_A20 IO_A21 IO_A22 IO_A23 IO_A24 IO_A25 IO_A26 IO_A27 IO_B20 IO_B21 IO_B22 IO_B23 IO_B24 IO_B26 IO_B25 IO_B27 IO_A13 IO_A12 PIAEROUT01 PIAEROUT02 COAEROUT PIC2001 PIC2002 COC20 PIC2101 PIC2102 COC21 PIC2201 PIC2202 COC22 PIC2301 PIC2302 COC23 PIC3801 PIC3802 COC38 PIC3901 PIC3902 COC39 PID00A PID00K COD0 PID10A PID10K COD1 PID20A PID20K COD2 PID30A PID30K COD3 PID40A PID40K COD4 PID50A PID50K COD5 PID60A PID60K COD6 PID70A PID70K COD7 PIJP1201 PIJP1202 PIJP1203 PIJP1204 PIJP1205 PIJP1206 PIJP1207 PIJP1208 PIJP1209 PIJP12010 PIJP12011 PIJP12012 PIJP12013 PIJP12014 PIJP12015 PIJP12016 PIJP12017 PIJP12018 PIJP12019 PIJP12020 COJP12 PILM1117010801 PILM1117010803 PILM1117010804 COLM11170108 PIP101 PIP102 PIP103 PIP104 PIP105 PIP106 PIP107 PIP108 PIP109 PIP1010 PIP1011 PIP1012 PIP1013 PIP1014 COP1 PIP201 PIP202 PIP203 PIP204 PIP205 PIP206 PIP207 PIP208 PIP209 PIP2010 PIP2011 PIP2012 PIP2013 PIP2014 PIP2015 PIP2016 PIP2017 PIP2018 PIP2019 PIP2020 PIP2021 PIP2022 PIP2023 PIP2024 PIP2025 PIP2026 PIP2027 PIP2028 PIP2029 PIP2030 PIP2031 PIP2032 PIP2033 PIP2034 COP2 PIP0GP101 PIP0GP102 PIP0GP103 PIP0GP104 PIP0GP105 PIP0GP106 PIP0GP107 PIP0GP108 PIP0GP109 PIP0GP1010 COP0GP1 PIPAB101 PIPAB102 PIPAB103 PIPAB104 PIPAB105 PIPAB106 PIPAB107 PIPAB108 PIPAB109 PIPAB1010 PIPAB1011 PIPAB1012 PIPAB1013 PIPAB1014 PIPAB1015 PIPAB1016 PIPAB1017 PIPAB1018 PIPAB1019 PIPAB1020 PIPAB1021 PIPAB1022 PIPAB1023 PIPAB1024 PIPAB1025 PIPAB1026 PIPAB1027 PIPAB1028 PIPAB1029 PIPAB1030 PIPAB1031 PIPAB1032 PIPAB1033 PIPAB1034 PIPAB1035 PIPAB1036 PIPAB1037 PIPAB1038 PIPAB1039 PIPAB1040 PIPAB1041 PIPAB1042 PIPAB1043 PIPAB1044 PIPAB1045 PIPAB1046 PIPAB1047 PIPAB1048 PIPAB1049 PIPAB1050 PIPAB1051 PIPAB1052 PIPAB1053 PIPAB1054 PIPAB1055 PIPAB1056 PIPAB1057 PIPAB1058 PIPAB1059 PIPAB1060 COPAB1 PIPAB201 PIPAB202 PIPAB203 PIPAB204 COPAB2 PIPB001 PIPB002 PIPB003 PIPB004 COPB0 PIPB101 PIPB102 PIPB103 PIPB104 COPB1 PIPB201 PIPB202 PIPB203 PIPB204 COPB2 PIPB301 PIPB302 PIPB303 PIPB304 COPB3 PIPB401 PIPB402 PIPB403 PIPB404 COPB4 PIPCD101 PIPCD102 PIPCD103 PIPCD104 PIPCD105 PIPCD106 PIPCD107 PIPCD108 PIPCD109 PIPCD1010 PIPCD1011 PIPCD1012 PIPCD1013 PIPCD1014 PIPCD1015 PIPCD1016 PIPCD1017 PIPCD1018 PIPCD1019 PIPCD1020 PIPCD1021 PIPCD1022 PIPCD1023 PIPCD1024 PIPCD1025 PIPCD1026 PIPCD1027 PIPCD1028 PIPCD1029 PIPCD1030 PIPCD1031 PIPCD1032 PIPCD1033 PIPCD1034 PIPCD1035 PIPCD1036 PIPCD1037 PIPCD1038 PIPCD1039 PIPCD1040 PIPCD1041 PIPCD1042 PIPCD1043 PIPCD1044 PIPCD1045 PIPCD1046 PIPCD1047 PIPCD1048 PIPCD1049 PIPCD1050 PIPCD1051 PIPCD1052 PIPCD1053 PIPCD1054 PIPCD1055 PIPCD1056 PIPCD1057 PIPCD1058 PIPCD1059 PIPCD1060 COPCD1 PIPCD201 PIPCD202 PIPCD203 PIPCD204 COPCD2 PIPVSUP01 PIPVSUP02 COPVSUP PIR101 PIR102 COR1 PIR201 PIR202 COR2 PIR301 PIR302 COR3 PIR2501 PIR2502 COR25 PIR2601 PIR2602 COR26 PIR2701 PIR2702 COR27 PIR2801 PIR2802 COR28 PIR2901 PIR2902 COR29 PIRL001PIRL002 CORL0 PIRL101PIRL102 CORL1 PIRL201PIRL202 CORL2 PIRL301PIRL302 CORL3 PIRL401 PIRL402 CORL4 PIRL501 PIRL502 CORL5 PIRL601 PIRL602 CORL6 PIRL701 PIRL702 CORL7 PISPINN01 PISPINN02 COSPINN PIU101 PIU102 PIU103 PIU104 PIU105 PIU106 PIU107 PIU108 PIU109 PIU1010 PIU1011 PIU1012 PIU1013 PIU1014 PIU1015 PIU1016 PIU1017 PIU1018 PIU1019 PIU1020 COU1 PIU201 PIU202 PIU203 PIU204 PIU205 PIU206 PIU207 PIU208 PIU209 PIU2010PIU2011 PIU2012 PIU2013 PIU2014 PIU2015 PIU2016 PIU2017 PIU2018 PIU2019 PIU2020 COU2 PIC3901 PILM1117010804 PIU1020 PIU201 PIP0GP102 PIPAB1029 PIPAB1030 PIPAB1033 PIPAB1034 PIPCD1031 PIPCD1032 PIPCD1035 PIPCD1036 PIR101 PIR201 PIR2501 PIR2701 PIU101 PIU2020 PIP2016 PIU209 NLACK0IN PIP2019 PIU1012 NLACK0OUT PIP2014 PIU1013 NLDATA0IN00 PIP2012 PIU1014 NLDATA0IN01 PIP2010 PIU1015 NLDATA0IN02 PIP208 PIU1016 NLDATA0IN03 PIP206 PIU1017 NLDATA0IN04 PIP204 PIU1018 NLDATA0IN05 PIP202 PIU1019 NLDATA0IN06 PIP2021 PIU202 NLDATA0OUT00 PIP2023 PIU203 NLDATA0OUT01 PIP2025 PIU204 NLDATA0OUT02 PIP2027 PIU205 NLDATA0OUT03 PIP2029 PIU206 NLDATA0OUT04 PIP2031 PIU207 NLDATA0OUT05 PIP2033 PIU208 NLDATA0OUT06 PIAEROUT01 PIC2002 PIC2102 PIC2202 PIC2302 PIC3802 PIC3902 PID00K PID10K PID20K PID30K PID40K PID50K PID60K PID70K PIJP12017 PIJP12018 PILM1117010801 PIP101 PIP103 PIP105 PIP107 PIP109 PIP1011 PIP1013 PIP201 PIP203 PIP205 PIP207 PIP209 PIP2011 PIP2013 PIP2015 PIP2017 PIP2018 PIP2020 PIP2022 PIP2024 PIP2026 PIP2028 PIP2030 PIP2032 PIP2034 PIP0GP101 PIPAB103 PIPAB104 PIPAB1031 PIPAB1032 PIPAB204 PIPB002 PIPB102 PIPB202 PIPB302 PIPB402 PIPCD103 PIPCD104 PIPCD1033 PIPCD1034 PIPCD203 PIPCD204 PISPINN01 PIU1010 PIU1011 PIU2010PIU2011 PIPAB105 PIRL002 NLIO0A3 NLLED00 PIPAB107 PIRL202 NLIO0A4 NLLED02 PIPAB109 NLIO0A5 PIPAB1011 NLIO0A6 PIPAB1013 NLIO0A7 PIP0GP103 PIPAB1015 NLGP100 NLIO0A8 PIP0GP105 PIPAB1017 NLGP102 NLIO0A9 PIP0GP107 PIPAB1019 NLGP104 NLIO0A10 PIP0GP109 PIPAB1021 NLGP106 NLIO0A11 PIAEROUT02 PIPAB1023 NLE0AEROUT NLIO0A12 PIPAB1025 PISPINN02 NLE0SPINN NLIO0A13 PIPAB1027 NLIO0A14 PIPAB1035 PIR2601 NLIO0A18 NLPRE0P PIPAB1037 PIR301 NLIO0A19 NLPRE0N PIPAB1039 PIU2018 NLIO0A20 PIPAB1041 PIU2016 NLIO0A21 PIPAB1043 PIU2014 NLIO0A22 PIPAB1045 PIU2012 NLIO0A23 PIPAB1047 PIU108 NLIO0A24 PIPAB1049 PIU106 NLIO0A25 PIPAB1051 PIU104 NLIO0A26 PIPAB1053 PIU102 NLIO0A27 PIPAB1055 NLIO0A28 PIPAB1057 NLIO0A29 PIPAB1059 NLIO0A30 PIPAB106 PIRL102 NLIO0B3 NLLED01 PIPAB108 PIRL302 NLIO0B4 NLLED04 PIPAB1010 NLIO0B5 PIPAB1012 NLIO0B6 PIPAB1014 NLIO0B7 PIP0GP104 PIPAB1016 NLIO0B8 NLPDM0DAT0L PIP0GP106 PIPAB1018 NLIO0B9 NLPDM0CLK0L PIP0GP108 PIPAB1020 NLIO0B10 NLPDM0CLK0R PIP0GP1010 PIPAB1022 NLIO0B11 NLPDM0DAT0R PIPAB1024 NLIO0B12 PIPAB1026 NLIO0B13 PIPAB1028 NLIO0B14 PIPAB1036 NLIO0B18 PIPAB1038 NLIO0B19 PIPAB1040 PIU2019 NLIO0B20 PIPAB1042 PIU2017 NLIO0B21 PIPAB1044 PIU2015 NLIO0B22 PIPAB1046 PIU2013 NLIO0B23 PIPAB1048 PIU109 NLIO0B24 PIPAB1050 PIU107 NLIO0B25 PIPAB1052 PIU105 NLIO0B26 PIPAB1054 PIU103 NLIO0B27 PIPAB1056 NLIO0B28 PIPAB1058 NLIO0B29 PIPAB1060 NLIO0B30 PIPCD105 PIRL602 NLIO0C3 NLLED07 PIPCD107 PIRL402 NLIO0C4 NLLED05 PIPCD109 NLIO0C5 PIPCD1011 NLIO0C6 PIPCD1013 NLIO0C7 PIPCD1015 NLIO0C8 PIPCD1017 NLIO0C9 PIPCD1019 NLIO0C10 PIPCD1021 NLIO0C11 PIPCD1023 NLIO0C12 PIPCD1025 NLIO0C13 PIPCD1027 NLIO0C14 PIPCD1029 NLIO0C15 PIPCD1037 NLIO0C19 PIJP1201 PIPCD1039 NLAER00 NLIO0C20 PIJP1203 PIPCD1041 NLAER02 NLIO0C21 PIJP1205 PIPCD1043 NLAER04 NLIO0C22 PIJP1207 PIPCD1045 NLAER06 NLIO0C23 PIJP1209 PIPCD1047 NLAER08 NLIO0C24 PIJP12011 PIPCD1049 NLAER010 NLIO0C25 PIJP12013 PIPCD1051 NLAER012 NLIO0C26 PIJP12015 PIPCD1053 NLAER014 NLIO0C27 PIJP12019 PIPCD1055 NLAER0REQ NLIO0C28 PIPCD1057 NLIO0C29 PIPCD1059 NLIO0C30 PIPCD106 PIRL702 NLIO0D3 NLLED08 PIPCD108 PIRL502 NLIO0D4 NLLED06 PIPCD1010 NLIO0D5 PIPCD1012 NLIO0D6 PIPCD1014 NLIO0D7 PIPCD1016 NLIO0D8 PIPCD1018 NLIO0D9 PIPCD1020 NLIO0D10 PIPCD1022 NLIO0D11 PIPCD1024 NLIO0D12 PIPCD1026 NLIO0D13 PIPCD1028 NLIO0D14 PIPCD1030 PIR2901 NLIO0D15 NLOUT0N PIPCD1038 PIR2801 NLIO0D19 NLOUT0P PIJP1202 PIPCD1040 NLAER01 NLIO0D20 PIJP1204 PIPCD1042 NLAER03 NLIO0D21 PIJP1206 PIPCD1044 NLAER05 NLIO0D22 PIJP1208 PIPCD1046 NLAER07 NLIO0D23 PIJP12010 PIPCD1048 NLAER09 NLIO0D24 PIJP12012 PIPCD1050 NLAER011 NLIO0D25 PIJP12014 PIPCD1052 NLAER013 NLIO0D26 PIJP12016 PIPCD1054 NLAER015 NLIO0D27 PIJP12020 PIPCD1056 NLAER0ACK NLIO0D28 PIPCD1058 NLIO0D29 PIPCD1060 NLIO0D30 PIP106 PIPAB202 NLJTAG0TCLK PIP1010 PIPAB201 NLJTAG0TDI PIP108 PIPCD201 NLJTAG0TDO PIP104 PIPCD202 NLJTAG0TMS PIP102 PIPAB203 NLJTAG0VIO PIC2001 PIPB001 PIR2502 PIR2602 PIC2101 PIPB101 PIR102 PIR302 PIC2201 PIPB201 PIR2702 PIR2802 PIC2301 PIPB301 PIR202 PIR2902 PID00A PIRL001 PID10A PIRL101 PID20A PIRL201 PID30A PIRL301 PID40A PIRL401 PID50A PIRL501 PID60A PIRL601 PID70A PIRL701 PIP1012 PIP1014 PIPB003 PIPB004 PIPB103 PIPB104 PIPB203 PIPB204 PIPB303 PIPB304 PIPB403 PIPB404 PIPB401 PIPCD102 NLRESET PIC3801 PILM1117010803 PIPCD101 PIPVSUP01 PIPAB101 PIPAB102 PIPVSUP02 FIGURE C.6: ZTEX 2.13 base board PCB schematic. 258 Appendix C. Generic purpose PCBs C.2.2 Board top: ZTEX_base_board.PcbDoc FIGURE C.7: ZTEX 2.13 base board top view. C.2. ZTEX 2.13 base board PCB files 259 C.2.3 Board bottom: ZTEX_base_board.PcbDoc PAAEROUT01 PAAEROUT02 COAEROUT PAC2001PAC2002 COC20 PAC2101PAC2102 COC21 PAC2201 PAC2202 COC22 PAC2301 PAC2302 COC23 PAC3801 PAC3802 COC38 PAC3901 PAC3902 COC39 PAD00APAD00K COD0 PAD10APAD10K COD1 PAD20APAD20K COD2 PAD30APAD30K COD3 PAD40A PAD40K COD4 PAD50A PAD50K COD5 PAD60A PAD60K COD6 PAD70A PAD70K COD7 PAJP1201 PAJP1202 PAJP1203 PAJP1204 PAJP1205 PAJP1206 PAJP1207 PAJP1208 PAJP1209 PAJP12010 PAJP12011 PAJP12012 PAJP12013 PAJP12014 PAJP12015 PAJP12016 PAJP12017 PAJP12018 PAJP12019 PAJP12020 COJP12 PALM1117010801 PALM1117010802 PALM1117010803 PALM1117010804 COLM11170108 PAP101 PAP103 PAP105 PAP107 PAP109 PAP1011 PAP1013 PAP102 PAP104 PAP106 PAP108 PAP1010 PAP1012 PAP1014 COP1 PAP2034 PAP2033 PAP2032 PAP2031 PAP2030 PAP2029 PAP2028 PAP2027 PAP2026 PAP2025 PAP2024 PAP2023 PAP2022 PAP2021 PAP2020 PAP2019 PAP2018 PAP2017 PAP2016 PAP2015 PAP2014 PAP2013 PAP2012 PAP2011 PAP2010 PAP209 PAP208 PAP207 PAP206 PAP205 PAP204 PAP203 PAP202 PAP201 COP2 PAP0GP1010 PAP0GP109 PAP0GP108PAP0GP107 PAP0GP106PAP0GP105 PAP0GP104PAP0GP103 PAP0GP102PAP0GP101 COP0GP1 PAPAB1051 PAPAB1052 PAPAB1053 PAPAB1054 PAPAB1055 PAPAB1056 PAPAB1057 PAPAB1058 PAPAB1059 PAPAB1060 PAPAB101 PAPAB102 PAPAB103 PAPAB104 PAPAB105 PAPAB106 PAPAB107 PAPAB108 PAPAB109 PAPAB1010 PAPAB1011 PAPAB1012 PAPAB1013 PAPAB1014 PAPAB1015 PAPAB1016 PAPAB1017 PAPAB1018 PAPAB1019 PAPAB1020 PAPAB1021 PAPAB1022 PAPAB1023 PAPAB1024 PAPAB1025 PAPAB1026 PAPAB1027 PAPAB1028 PAPAB1029 PAPAB1030 PAPAB1031 PAPAB1032 PAPAB1033 PAPAB1034 PAPAB1035 PAPAB1036 PAPAB1037 PAPAB1038 PAPAB1039 PAPAB1040 PAPAB1041 PAPAB1042 PAPAB1043 PAPAB1044 PAPAB1045 PAPAB1046 PAPAB1047 PAPAB1048 PAPAB1049 PAPAB1050 COPAB1 PAPAB204PAPAB203 PAPAB202PAPAB201 COPAB2 PAPB004 PAPB003 PAPB002 PAPB001 COPB0 PAPB104 PAPB103 PAPB102 PAPB101 COPB1 PAPB204 PAPB203 PAPB202 PAPB201 COPB2 PAPB304 PAPB303 PAPB302 PAPB301 COPB3 PAPB404 PAPB403 PAPB402 PAPB401 COPB4 PAPCD1051 PAPCD1052 PAPCD1053 PAPCD1054 PAPCD1055 PAPCD1056 PAPCD1057 PAPCD1058 PAPCD1059 PAPCD1060 PAPCD101 PAPCD102 PAPCD103 PAPCD104 PAPCD105 PAPCD106 PAPCD107 PAPCD108 PAPCD109 PAPCD1010 PAPCD1011 PAPCD1012 PAPCD1013 PAPCD1014 PAPCD1015 PAPCD1016 PAPCD1017 PAPCD1018 PAPCD1019 PAPCD1020 PAPCD1021 PAPCD1022 PAPCD1023 PAPCD1024 PAPCD1025 PAPCD1026 PAPCD1027 PAPCD1028 PAPCD1029 PAPCD1030 PAPCD1031 PAPCD1032 PAPCD1033 PAPCD1034 PAPCD1035 PAPCD1036 PAPCD1037 PAPCD1038 PAPCD1039 PAPCD1040 PAPCD1041 PAPCD1042 PAPCD1043 PAPCD1044 PAPCD1045 PAPCD1046 PAPCD1047 PAPCD1048 PAPCD1049 PAPCD1050 COPCD1 PAPCD204PAPCD203 PAPCD202PAPCD201 COPCD2 PAPVSUP02 PAPVSUP01 COPVSUP PAR101PAR102 COR1 PAR201 PAR202 COR2 PAR301PAR302 COR3 PAR2501 PAR2502 COR25 PAR2601 PAR2602 COR26 PAR2701 PAR2702 COR27 PAR2801 PAR2802 COR28 PAR2901 PAR2902 COR29 PARL001 PARL002 CORL0 PARL101 PARL102 CORL1 PARL201 PARL202 CORL2 PARL301 PARL302 CORL3 PARL401PARL402 CORL4 PARL501PARL502 CORL5 PARL601PARL602 CORL6 PARL701 PARL702 CORL7 PASPINN01 PASPINN02 COSPINN PAU101 PAU102 PAU103 PAU104 PAU105 PAU106 PAU107 PAU108 PAU109 PAU1010 PAU1020 PAU1019 PAU1018 PAU1017 PAU1016 PAU1015 PAU1014 PAU1013 PAU1012 PAU1011 COU1 PAU201 PAU202PAU203PAU204 PAU205PAU206PAU207PAU208PAU209 PAU2010 PAU2020 PAU2019PAU2018PAU2017 PAU2016PAU2015PAU2014PAU2013PAU2012 PAU2011 COU2 PAC3901 PALM1117010804 PAU1020 PAU201 PAP0GP102 PAPAB1029 PAPAB1030 PAPAB1033 PAPAB1034 PAPCD1031 PAPCD1032 PAPCD1035 PAPCD1036 PAR101 PAR201 PAR2501 PAR2701 PAU101 PAU2020 PAP2016 PAU209 PAP2019 PAU1012 PAJP1201 PAPCD1039 PAJP1202 PAPCD1040 PAJP1203 PAPCD1041 PAJP1204 PAPCD1042 PAJP1205 PAPCD1043 PAJP1206 PAPCD1044 PAJP1207 PAPCD1045 PAJP1208 PAPCD1046 PAJP1209 PAPCD1047 PAJP12010 PAPCD1048 PAJP12011 PAPCD1049 PAJP12012 PAPCD1050 PAJP12013 PAPCD1051 PAJP12014 PAPCD1052 PAJP12015 PAPCD1053 PAJP12016 PAPCD1054 PAJP12020 PAPCD1056 PAJP12019 PAPCD1055 PAP2014 PAU1013 PAP2012 PAU1014 PAP2010 PAU1015 PAP208 PAU1016 PAP206 PAU1017 PAP204 PAU1018 PAP202 PAU1019 PAP2021 PAU202 PAP2023 PAU203 PAP2025 PAU204 PAP2027 PAU205 PAP2029 PAU206 PAP2031 PAU207 PAP2033 PAU208 PAAEROUT02 PAPAB1023 PAPAB1025 PASPINN02 PAAEROUT01 PAC2002 PAC2102 PAC2202 PAC2302 PAC3802 PAC3902 PAD00K PAD10K PAD20K PAD30K PAD40K PAD50K PAD60K PAD70K PAJP12017 PAJP12018 PALM1117010801 PAP101 PAP103 PAP105 PAP107 PAP109 PAP1011 PAP1013 PAP201 PAP203 PAP205 PAP207 PAP209 PAP2011 PAP2013 PAP2015 PAP2017 PAP2018 PAP2020 PAP2022 PAP2024 PAP2026 PAP2028 PAP2030 PAP2032 PAP2034 PAP0GP101 PAPAB103 PAPAB104 PAPAB1031 PAPAB1032 PAPAB204 PAPB002 PAPB102 PAPB202 PAPB302 PAPB402 PAPCD103 PAPCD104 PAPCD1033 PAPCD1034 PAPCD203 PAPCD204 PASPINN01 PAU1010 PAU1011 PAU2010 PAU2011 PAP0GP103 PAPAB1015 PAP0GP105 PAPAB1017 PAP0GP107 PAPAB1019 PAP0GP109 PAPAB1021 PAPAB105 PARL002 PAPAB107 PARL202 PAPAB109 PAPAB1011 PAPAB1013 PAPAB1027 PAPAB1035 PAR2601 PAPAB1037 PAR301 PAPAB1039 PAU2018 PAPAB1041 PAU2016 PAPAB1043 PAU2014 PAPAB1045 PAU2012 PAPAB1047 PAU108 PAPAB1049 PAU106 PAPAB1051 PAU104 PAPAB1053 PAU102 PAPAB1055 PAPAB1057 PAPAB1059 PAPAB106 PARL102 PAPAB108 PARL302 PAPAB1010 PAPAB1012 PAPAB1014 PAP0GP104 PAPAB1016 PAP0GP106 PAPAB1018 PAP0GP108 PAPAB1020 PAP0GP1010 PAPAB1022 PAPAB1024 PAPAB1026 PAPAB1028 PAPAB1036 PAPAB1038 PAPAB1040 PAU2019 PAPAB1042 PAU2017 PAPAB1044 PAU2015 PAPAB1046 PAU2013 PAPAB1048 PAU109 PAPAB1050 PAU107 PAPAB1052 PAU105 PAPAB1054 PAU103 PAPAB1056 PAPAB1058 PAPAB1060 PAPCD105 PARL602 PAPCD107 PARL402 PAPCD109 PAPCD1011 PAPCD1013 PAPCD1015 PAPCD1017 PAPCD1019 PAPCD1021 PAPCD1023 PAPCD1025 PAPCD1027 PAPCD1029 PAPCD1037 PAPCD1057 PAPCD1059 PAPCD106 PARL702 PAPCD108 PARL502 PAPCD1010 PAPCD1012 PAPCD1014 PAPCD1016 PAPCD1018 PAPCD1020 PAPCD1022 PAPCD1024 PAPCD1026 PAPCD1028 PAPCD1030 PAR2901 PAPCD1038 PAR2801 PAPCD1058 PAPCD1060 PAP106 PAPAB202 PAP1010 PAPAB201 PAP108 PAPCD201 PAP104 PAPCD202 PAP102 PAPAB203 PAC2001 PAPB001 PAR2502 PAR2602 PAC2101 PAPB101 PAR102 PAR302 PAC2201 PAPB201 PAR2702PAR2802 PAC2301 PAPB301 PAR202 PAR2902 PAD00A PARL001 PAD10A PARL101 PAD20A PARL201 PAD30A PARL301 PAD40A PARL401 PAD50A PARL501 PAD60A PARL601 PAD70A PARL701 PAPB401 PAPCD102 PAC3801 PALM1117010803 PAPCD101 PAPVSUP01 PAPAB101 PAPAB102 PAPVSUP02 FIGURE C.8: ZTEX 2.13 base board bottom view. 260 Appendix C. Generic purpose PCBs FIGURE C.9: ZTEX 2.13 base board bottom 3D view.