Live Demonstration: Real-time motor rotation frequency detection by spike-based visual and auditory AER sensory integration for FPGA

Live Demonstration: Real-time motor rotation frequency detection by spike-based visual and auditory AER sensory integration for FPGA A. Rios-Navarro, E. Cerezuela-Escudero, M. Dominguez-Morales, A. Jimenez-Fernandez, G. Jimenez-Moreno, A. Linares-Barranco Robotic and Technology of Computers Lab. University of Seville Seville, Spain Email: [email protected] Abstract- Multisensory integration is commonly used in various robotic areas to collect much more information from an environment using different and complementary types of sensors. This demonstration presents ascenario where the motor rotation frequency is obtained using an AER DVS128 retina chip (Dynamic Vision Sensor) and afrequency decomposer auditory system on aFPGA that mimics abiological cochlea. Both of them are spike-based sensors with Address-Event-Representation (AER) outputs. Anew AER monitor hardware interface, based on aSpartan-6 FPGA, allows two operational modes: real-time (up to 5Mevps through USB2.0) and off-line mode (up to 20Mevps and 33.5Mev stored in DDR RAM). The sensory integration allows the bio-inspired cochlea limit to provide a concrete range of rpm approaches, which are obtained by the silicon retina. I. INTRODUCTION This demonstration consists of the system shown in the photo (Fig. 1)[ 1]. Acircle of metal with ashape drawn is mounted on aDC motor that will be rotating while DVS and bio-inspired cochlea are capturing information from the scene. The two spike-based sensors information are transmitted on board where both are merged and captured. Two operation modes are implemented in this controller board: real-time, in which all information received from the sensor is time stamped and sent to the PC through USB 2.0 board interface; and offline, where sensor information is stored in DDR2 RAM memory, together with precise temporal information, allowing later processing in aPC by downloading it from DDR2. II. HARDWARE INVOLVED The components used in this demonstration are: aDC motor with ametal circle; aDVS128 silicon retina; abioinspired auditory system (microphone, equalizer and Virtex5 development board); and the XEM 6010 Opal Kelly board. The event merger and monitor/logger circuit captures this information. It is implemented on the Opal Kelly Spartan 6 FPGA. As discussed above, monitor/logger circuit allows two operations mode: real-time and off-line. In real-time mode, the AER data stream from the both sensor is time stamped and sent to aPC directly through aUSB This work has been supported by BIOSENSE (TEC2012-37868-C04- 02/01) 2.0 interface. The maximum amount of information captured is limit up 5Mevps. This is due to the bottleneck of the USB 2.0. In off-line mode the maximum amount of information captured by the system is higher (about 20 Mevps) because the events are time-stamped and stored immediately in DDR2 memory where the FPGA-DDR2 interface is faster than USB 2.0 interface. For both methods the monitor/logger circuit sends the information to the PC where it is processed under jAER software application. In this software application, the AER DVS128 sensor information is used by afilter to estimate the motor speed while the frequency decomposer auditory system information is processed by aConvolutional Neural Network (CNN) to get avalid range of motor speed. The output of AER DVS128 filter that be out of this valid range, will be rejected in order to calculate de final system output. Fig. 1. Completed setup with DC motor, bio-inspired cochlea system (up rigth comer), monitor/logger system (down right comet) and DVS128 silicon retina (middle). III. VISITORS EXPERIENCE Visitors can interact with the system by changing, through a microcontroller, the motor rotation frequency while sensors are capturing events to estimate the motor frequency. REFERENCE [1] A. Rios-Navarro, A. Jimenez-Fernandez, E. Cerezuela-Escudero, M. Rivas, G. Jimenez-Moreno, A. Linares-Barranco. "Live Demonstration: Real-Time Motor Rotation Frequency Detection by Spike-Based Visual and Auditory Sensory Fusion on AER and FPGA" LNCS 8681, pp 847-848 ISSN 0302-9743 ISBN 978-3-319- 11178-0 978-1-4799-8391-9/15/$31.00 ©20 15 IEEE 1907