Towards the Neuromorphic Implementation of the Auditory Perception in the iCub Robotic Platform
Abstract
Hearing can be considered as one of the most important senses since it plays a key role in the audiovisual learning process. While a lot of effort has been made for achieving good results from the traditional approach of auditory perception, new trends such as neuromorphic computing are showing promising achievements in the implementation of brain structures for sensory perception. In this work, the design and integration of a neuromorphic event based digital model of the auditory ascending pathway within the iCub robotic platform is proposed. This model, which comprises from the cochlea up to the inferior colliculus, replaces the traditional approach for sound processing already implemented on iCub and is able to perform sound recognition and spatial localization in real time, allowing the implementation of auditory attention models in complex scenarios, like the cocktail party problem
Full text
Towa ds he Neu omo phic Implemen a ion o he Audi o y
Pe cep ion in he iCub Robo ic Pla o m
Daniel Gu ie ez-Galan
Uni e si y o Se illa
Se illa, Spain
[email p o ec ed]
Chia a Ba olozzi
Is i u o I aliano di Tecnologia
Genoa, I aly
Juan Ped o
Dominguez-Mo ales
Uni e si y o Se ille
Se ille, Spain
Angel Jimenez-Fe nandez
Uni e si y o Se ille
Se ille, Spain
Alejand o Lina es-Ba anco
Uni e si y o Se ille
Se ille, Spain
ABSTRACT
Hea ing can be conside ed as one o he mos impo an senses
since i plays a key ole in he audio isual lea ning p ocess. While
a lo o e o has been made o achie ing good esul s om he
adi ional app oach o audi o y pe cep ion, new ends such as
neu omo phic compu ing a e showing p omising achie emen s
in he implemen a ion o b ain s uc u es o senso y pe cep ion.
In his wo k, he design and in eg a ion o a neu omo phic e en -
based digi al model o he audi o y ascending pa hway wi hin he
iCub obo ic pla o m is p oposed. This model, which comp ises
om he cochlea up o he in e io colliculus, eplaces he adi ional
app oach o sound p ocessing al eady implemen ed on iCub and
is able o pe o m sound ecogni ion and spa ial localiza ion in eal-
ime, allowing he implemen a ion o audi o y a en ion models in
complex scena ios, like he cock ail pa y p oblem.
CCS CONCEPTS
•Compu ing me hodologies → Bio-inspi ed app oaches
;
•␣
Applied compu ing → E en -d i en a chi ec u es.
KEYWORDS
neu omo phic audi o y senso , iCub, e en -based p ocessing, sound
ecogni ion, sound localiza ion
1 INTRODUCTION
Humanoid obo s will soon be a common p esence among humans
since hei capabili ies o in e ac ion wi h he en i onmen and
humans a e s eadily inc easing, wi h objec and human ecogni ion
[
9
], and speech unde s anding [
5
]. Howe e , hese capabili ies com-
monly esul in a high compu a ional cos and a e o en deployed in
emo e compu ing sys ems, wi h he need o cons an da a ans e .
Neu omo phic pe cep ion and compu a ion, inspi ed by he compu-
a ional p inciples o biological neu al sys ems, can be used owa ds
low-powe , embedded solu ions o complex pe cep i e asks. In his
wo k, a digi al e en -based bio-inspi ed model [
6
] o he Audi o y
Ascending Pa hway (
AAP
) [
1
] on he iCub humanoid pla o m [
7
]
was implemen ed as a neu omo phic al e na i e o he adi ional
audi o y models in o de o educe la ency, powe consump ion
and compu a ional cos . This model, which has been alida ed o
sound ecogni ion and sound sou ce localiza ion, can be ex ended o
implemen mo e complex audi o y p ocessing and audio- isual u-
sion [
10
], in o de o imp o e he accu acy and lea ning capabili ies
o he sys em.
2 GIVING ICUB THE SENSE OF HEARING
The audi o y model comp ises he Neu omo phic Audi o y Com-
plex (
NAC
), implemen ed on he Zynq Field P og ammable Ga e
A ay (
FPGA
) o econ igu abili y and eal- ime signal p ocessing,
and he In e io Colliculus (
IC
) on SpiNNake , whe e la ge b ain
a eas can be implemen ed, as shown in Fig. 1(A). Thanks o he bidi-
ec ional communica ion be ween iCub and SpiNNake [
8
] h ough
Ye Ano he Robo Pla o m (
YARP
) [
2
], eal- ime, close-loop audio
applica ions can be implemen ed. The
NAC
module was in eg a ed
wi hin he iCub obo ic pla o m as an In ellec ual P ope y (
IP
)
co e connec ed o he Head P ocessing Uni (
HPU
) module. The
NAC
ecei es sound inpu om he wo mic ophones placed in
he ea s o he iCub. I is composed o h ee sub-modules: 1) a 32-
channels binau al Neu omo phic Audi o y Senso (
NAS
) [
6
], which
implemen s an e en -based digi al cochlea model; 2) he Spike-
based Supe io Oli e (
SSO
), which implemen s bo h he In e au al
Time Di e ence (
ITD
) [
3
] and he In e au al Le el Di e ence (
ILD
)
ex ac ion om he
NAS
’ ou pu e en s; and 3) he e en s moni o
oge he wi h he in e ace o he
HPU
co e, which is ca ied ou by
using he Add ess E en Rep esen a ion (
AER
) p o ocol. The sound
ea u es ex ac ed wi h he
NAC
a e combined and in eg a ed by a
mul ilaye Spiking Neu al Ne wo k (
SNN
) in SpiNNake ha mod-
els he beha io o he
IC
[
1
] o ex ac he ele an in o ma ion
ega ding he loca ion o he objec in he ho izon al axis. Then, he
ou pu o he
SNN
is sen back o he iCub o u n i s head owa ds
he sound sou ce. This close-loop sys em allows eal- ime sound
I2S
mic ophone
ZynQ
NAC
HPU Co e
SpiNNake
SNNs
Le
mic.
Le
came a
Righ
mic.
Righ
came a
Compu e
B)A) C)
Figu e 1: A) iCub-NAC in eg a ion diag am. B) Ou pu e en s om he localiza ion model. C) Sonog am om he keywo d
spo ing.
sou ce localiza ion and is he basic building block o audi o y
a en ion [4].
P elimina y esul s o he
NAC
-iCub in eg a ion show ha iCub
was success ully able o localize a sound sou ce which was shi ed
om le o igh and ice e sa in on o he iCub, as shown in
Fig. 1B). In addi ion, a da ase o spoken digi s, shown in Fig. 1C),
was di ec ly eco ded om he iCub o ain a
SNN
o pe o ming
keywo d spo ing. To he bes o ou knowledge, his is he i s
ime ha a digi al neu omo phic audi o y model has been in e-
g a ed wi hin a humanoid obo , and hese esul s open he doo
o u u e implemen a ions o senso y usion models wi h lea ning
algo i hms o sol e asks such as he cock ail pa y p oblem on a
ully neu omo phic eal- ime obo ic pla o m, bene i ing om he
compu a ional la ency and low-powe consump ion na u e o hese
sys ems.
ACKNOWLEDGMENTS
This wo k was suppo ed by he Spanish g an MINDROB (PID2019-
105556GB-C33/AEI/10.13039/501100011033), and SMALL (PCI2019-
111841-2/AEI/10.1309/501100011033 om EU CHIST-ERA p og am).
Pa o his wo k was ca ied ou du ing a esea ch in e nship
o D. G.-G. in he EDPR g oup (I alian Ins i u e o Technology,
Genoa (I aly)) suppo ed by a Fo mación de Pe sonal In es igado
Schola ship om he Spanish Minis y o Educa ion and Cul u e.
REFERENCES
[1]
Jo ge Dá ila-Chacón e al
.
2018. Enhanced obo speech ecogni ion using
biomime ic binau al sound sou ce localiza ion. IEEE ansac ions on neu al ne -
wo ks and lea ning sys ems 30, 1 (2018), 138–150.
[2]
A en Glo e e al
.
2018. The e en -d i en so wa e lib a y o YARP—Wi h
algo i hms and iCub applica ions. F on ie s in Robo ics and AI 4 (2018), 73.
[3]
Daniel Gu ie ez-Galan e al
.
2019. A neu omo phic app oach o he sound
sou ce localiza ion ask in eal- ime embedded sys ems: wo k-in-p og ess. In
P oceedings o he In e na ional Con e ence on Embedded So wa e Companion.
1–2.
[4]
Dillon A Hamb ook e al
.
2017. A Bayesian compu a ional basis o audi o y
selec i e a en ion using head o a ion and he in e au al ime-di e ence cue.
PloS one 12, 10 (2017).
[5]
Be and Higy, Alessio Me e a, Gio gio Me a, and Leona do Badino. 2018. Speech
ecogni ion o he icub pla o m. F on ie s in Robo ics and AI 5 (2018), 10.
[6]
Angel Jiménez-Fe nández e al
.
2017. A Binau al Neu omo phic Audi o y Senso
o FPGA: A Spike Signal P ocessing App oach. IEEE T ans. Neu al Ne w. Lea ning
Sys . 28, 4 (2017), 804–818.
[7]
Lo enzo Na ale e al
.
2017. icub: The no -ye - inished s o y o building a obo
child. Science Robo ics 2, 13 (2017).
[8]
Eus ace Paink as e al
.
2013. SpiNNake : A 1-W 18-co e sys em-on-chip o
massi ely-pa allel neu al ne wo k simula ion. IEEE Jou nal o Solid-S a e Ci cui s
48, 8 (2013), 1943–1953.
[9]
Giulia Pasquale, Ca lo Cilibe o, F ancesca Odone, Lo enzo Rosasco, and Lo enzo
Na ale. 2019. A e we done wi h objec ecogni ion? The iCub obo ’s pe spec i e.
Robo ics and Au onomous Sys ems 112 (2019), 260–281.
[10]
Vadim Tikhano e al
.
2010. In eg a ion o speech and ac ion in humanoid
obo s: iCub simula ion expe imen s. IEEE T ansac ions on Au onomous Men al
De elopmen 3, 1 (2010), 17–29.