Mul iplexing AER Asynch onous Channels
o e
LVDS
Links
wi h Flow-Con ol and Clock-Co ec ion o
Scalable
Neu omo phic Sys ems
A. Youse zadeh
1
, M. JabáoĔski
2
, T. Iakymchuk
3
, A. Lina es-Ba anco
4
, A. Rosado
3
, L.
A.
Plana
5
, T. Se ano-Go a edona
1
, S. Fu be
5
, and B. Lina es-Ba anco
1
1
Ins i u o de Mic oelec ónica de Se illa (CSIC and Uni . de Se illa), Se illa, Spain { eza, be nabe}@imse-cnm.csic.es
2
AGH Uni e si y o Science and Technology, K akow, Poland
3
School o Enginee ing (Uni . o Valencia), Spain
4
Dep . o Compu e A chi ec u es (Uni . o Se illa), Spain
5
Dep . Comp. Science, Uni e si y o Manches e , UK
Abs ac —
Add ess-E en -Rep esen a ion
(AER) is a widely
ex ended
asynch onous
echnique o
in e changing “neu al
spikes”
among di e en
ha dwa e
elemen s in
Neu omo phic
Sys ems.
Con en ional AER links use
pa allel
physical wi es
oge he wi h a pai o
handshaking signals
(Reques and
Acknowledge). He e we p esen a
ully se ial
implemen a ion
using
bidi ec ional
SATA connec o s
wi h
a pai o LVDS (low
ol age
di e en ial
signaling) wi es o
each
di ec ion. The
p oposed
implemen a ion
can mul iplex a
numbe
o
con en ional pa allel AER links pe LVDS physical
connec ion.
I uses low con ol, clock co ec ion, and by e alignmen
ech
niques o
ansmi
32-bi add ess e en s
eliably
o e
mul iplexed
se ial connec ions. The se up has been es ed using
comme
cial
Spa an6
FPGAs
eaching
a maximum e en
ansmission speed
o 75Meps (Mega E en s pe second) o
32-bi e en s a
3.0Gbps
line da a a e.
Keywo ds—
Neu omo phic
Sys ems, Vi ual Wi ing,
AER
(Add ess E en
Rep esen a ion),
Scalable
Neu omo phic
Sys ems.
I. I
NTRODUCTION
Add ess E en Rep esen a ion (AER) is a popula “ i ual
wi ing” echnique used by many neu omo phic ha dwa e
enginee s o in e connec spiking neu omo phic sys ems. As
neu omo phic sys ems ha e been scaling up in size and
complexi y o e he yea s, esea che s ha e de eloped mo e
complex and sma e AER “ a ia ions” o imp o e
e iciency,
econ igu abili y
and eliabili y. Since e y ea ly [1]
i became appa en ha he o iginal pa allel-AER (pAER) bus
bulkiness would se iously limi he scalabili y o AER
sys ems o a bi a y size, and esea che s s a ed o look a
se ial connec i i y op ions. Fully bi -se ial Low-Vol age-
Di e en ial-Signaling (LVDS) allow o mul i-gigabi -pe -
second communica ions wi h only a pai o wi es. Howe e ,
wi h only wo uni-di ec ional di e en ial wi es i is no
ob ious o implemen a handshaking p o ocol pe e en
ansmission, no a low-con ol scheme o signal da a
conges ion a he ecei e side. Be ge e al. [2] expe imen ed
wi h he 2.5Gbps (Giga-bi -pe -second) LVDS IP block
a ailable in he Vi exII-P o Xilinx FPGAs, achie ing
41.66Meps (Mega-e en s-pe -second) o 16-bi AEs
(Add ess E en s).
Howe e , he e was no hand-shaking no low-con ol
mechanism o a oid da a loss in case he ecei e side
would be empo a ily slowe han he ansmi e side.
Fasnach e al. [3] epo ed a bi -se ial in e ace based on o -
he-shel 16-bi Se ialize /Dese ialize comme cial
componen s (TLK 2501/3101) connec ed o a Spa an3E
pa allel e en AER p ocesso , using a second backwa ds
LVDS link o low con ol signaling. The bi -se ial link could
ope a e a 2.5Gbps (TLK 2501) o 3.125Gbps (TLK3101) line
speed and also used 8b/10b encoding o allow o idle
commas. On he LVDS backwa d link he ecei e pu s a
squa e wa e whose equency signals whe he o s op o
esume e en ansmission om he sende . Using his se up
he link could ansmi 32-bi e en s a a maximum a e o
62.5Meps ( o 2.5Gbps) o 78.125Meps ( o 3.125Gbps), a
he cos o sac i icing one LVDS backwa d link o low
con ol.
Zama en˜
o e al. [4] de eloped a bi-di ec ional LVDS
link using Vi ex6 FPGA Rocke -I/O IPs. In his scheme wo
2.5Gbps LVDS links a e used, each o communica ing 32-bi
e en s in each di ec ion wi h 8b/10b encoding. Flow-con ol
in each di ec ion is implemen ed by special con ol symbols
Fig. 1: Example se up o se en een Spa an6 PCBs communica ing
h ough bidi ec ional LVDS SATA o hogonal links, plus a DVS came as
and USB-AER boa d communica ing h ough pa allel AER buses wi h
one o he Spa an6 PCBs.
Fig. 2: Block Diag am o Bidi ec ional LVDS link communica ing wo pa allel AER asynch onous links o opposi e di ec ion
sen in he opposi e link di ec ion o oggle be ween he s op
and esume s a es. Howe e , his design did no include any
clock-co ec ion suppo , which limi s he app oach o he case
when all FPGAs use he same physical e e ence clock, hus
hampe ing scalabili y.
All p e ious asynch onous pAER o bi -se ial AER
con e sion schemes ope a e co ec ly i he e is only one clock
domain. Howe e , his is no iable when one wan s o
in e connec mul iple FPGAs, each wi h i s own local-clock
synch onous e en p ocessing subsys em, oge he wi h
mul iple bidi ec ional LVDS links pe FPGA. In his case,
each FPGA will ha e one o mo e local clock domains,
which can be in e e ing wi h he clock domains o
neighbo ing FPGAs. Unde hese ci cums ances, i is necessa y
o use some clock co ec ion echnique in o de o compensa e
o clock equency/phase d i s and a oid sudden by e
misalignmen p oblems and da a loss. In a p elimina y wo k
[5] we exploi ed he use o elas ic bu e s a ailable wi hin
Xilinx Rocke -I/O se ial LVDS IPs, al hough i was used
o ully synch onous handshake-less sys ems deployed o e
mul iple FPGAs. Each link was unidi ec ional since he
backwa d LVDS pa h was ully used o handling low
con ol. Mo e ecen ly, he SpiNNake eam has de eloped
bidi ec ional bi -se ial LVDS links [ 6 ] o bundle eigh 2-
o -7 AER mul i-symbol in e -SpiNNake -chip links [7] in o
one bi -se ial SATA link. This scheme has been de eloped o
in e connec mul iple (up o 1200) 48-chip SpiNNake Boa ds
[8]. The scheme uses low-con ol, clock co ec ion, oge he
wi h a complex aming p o ocol ha samples he eigh
channels and pe o ms CRC (Cyclic Redundancy Checks) o
inc ease eliabili y. This in oduces ex a o e heads, limi ing
he maximum h oughpu heo e ically o 50Meps (maximum
6.25Meps pe channel).
He e we p esen an ex ended e sion o he one epo ed
ea lie [5], bu wi h ully bidi ec ional bi -se ial LVDS
communica ion capabili y, wi h oken-based low-con ol
p o ocol oge he wi h clock co ec ion capabili y, as well as
a obus in e ace o con en ional pa allel AER po s (like
hose used in AER senso chips) wi h 4-phase handshaking
asynch onous communica ion. Using a 3.0Gbps LVDS line
ansmission a e i is possible o achie e 32-bi 75.0Meps
sus ained ansmission in each di ec ion o he link. Fig. 1
shows an example a ge se up consis ing o an a ay o 17
AER-Node Boa ds [ 5 ] in e connec ed h ough SATA o hei
neighbo s. Addi ionally, an AER e ina senso and a USB-
AER boa d [13] a e connec ed o he se up h ough pa allel
AER buses [9].
II. B
IDIRECTIONAL
LVDS
AER
B
IT
-S
ERIAL
L
INK WITH
F
LOW
-C
ONTROL
,
C
LOCK
C
ORRECTION AND
B
YTE
-A
LIGNMENT
8b/10b encoding [10] ans o ms 8-bi by es in o 10-bi
wo ds while dc balancing ze os and ones. Ex a dc-balanced
10-bi wo ds a e a ailable, which can be used by he use as
“command” cha ac e s, also called “k-cha s” o “commas”.
He e we use some o hese k-cha s o pe o m low con ol,
by e alignmen s and clock co ec ions.
Fig. 2 shows a block diag am o a bidi ec ional AER
link using wo FPGA PCBs connec ed h ough a single SATA
cable (con aining wo pai s o wi es). Each FPGA connec s o
one hand-shaken 32-bi pAER sende and one pAER ecei e .
The “w appe ” block con ains a w appe ansmi e sub-block
wTX and a w appe ecei e sub-block wRX. wTX akes as
inpu a 32-bi clock-synch onous “DATA” wo d and a 4-bi
“k-cha ” wo d. Each o he ou “k-cha ” bi s indica e whe he
he co esponding ou by es in he 32-bi “DATA” wo d a e
ei he a con ol comma by e o a egula use da a by e. In ou
case, LVDS line a es we e 3.0Gbps, ob ained om a low ji e
di e en ial 150MHz e e ence X al oscilla o on he FPGA
PCB. This means ha he 32-bi e en da a, ans o med in o a
40-bi sequence by 8b/10b encoding, needs
40
/
(3
×
10
9
Hz)
=
13.3ns o be ansmi ed. The w appe p o ides wo
e e ence clocks o he use ci cui y. One a equency
=
1
/
13
.
3
ns
=
75.0MHz, a he ising edges o which he
use ci cui y has o p o ide he 32-bi pa allel “DATA” wo d
and he co esponding 4-bi “k-cha ” wo d. The o he e e ence
clock has equency 4 = 300MHz (because he 32-bi
“DATA” wo d con ains 4 by es) and is synch onized o clock
.La ency o ansmission in ou se up is 20 clock cycles
(266ns). The use designed ci cui y wi hin each FPGA in Fig.
2 comp ises 3 blocks:
1)
T ansmi e
Block (TX). This block handles he
asynch onous handshaking wi h he inpu 32-bi AERin po ,
he synch oniza ion be ween he asynch onous and
synch onous domains and “S op/Run” s a e o he
ansmission o da a which is used by he low con ol
p o ocol. Also, his block p o ides o each clock cycle he 32-
bi “DATA” wo d and he 4-bi “k-cha ” wo d equi ed by he
“w appe ” (wTX), gene a es s a up by e alignmen sequences,
and inse s con ol symbols o low con ol, clock co ec ion,
idle commas, and pe iodic commas o alignmen .
2) Recei e Block (RX). This block ecei es and sepa a es
da a and con ol commas. Da a symbols a e sen o he FIFO
block, while con ol commas a e in e p e ed and execu ed o
p ope low con ol, wo d ( e)alignmen , and clock co ec ion.
3) FIFO block. This block accumula es 32-bi synch onous
da a symbols om he RX block in o a FIFO egis e , while i
emp ies he FIFO by sending da a ou o he AERo po . This
block handles he asynch onous handshaking wi h he ou pu
32-bi AERo po and he synch oniza ion wi h he
synch onous da a “DATA” clock domain. I he FIFO ge s
illed up abo e a h eshold i will igge he low con ol
mechanism.
Fig. 2 illus a es low con ol mechanism. I FIFO block is
close o ge ing ull, i sends a 4-by e s op oken con ol comma
h ough he backwa d link o i s co esponding TX block on he
ansmi e side, se ing i in o ‘s op’ mode. In his mode Ack
will no be acknowledged and he pa allel AER po s ops
aking new e en s. Once he FIFO block has enough ee space,
a 4-by e esume oken is sen in he same manne o disable TX
block s op mode and esume communica ion. Fo clock
co ec ion, pe iodic 4-by e con ol commas a e inse ed in he
da a low by he ansmi e . I he ecei e clock is
as e /slowe i will e ase/inse such commas h ough i s
elas ic bu e , hus equalizing he e ec i e da a a e be ween
ansmi e and ecei e side. Addi ionally, when no da a a e
sen , 4-by e idle commas a e inse ed o keep he LVDS link
unning and aligned.
III. M
ULTIPLE
AER
C
HANNELS
M
ULTIPLEXING
Fo mul iple AER channels mul iplexing he app oach
shown in Fig. 3 is used. The op q bi s o he 32-bi da a wo d
a e used o encode k AER channels (2
q
 k). Each AER channel
TX-FIFO eques s access o an encode in he CH-MUX block.
Once access is g an ed, communica ion goes h ough he CH-
MUX in a simila way as was desc ibed in he p e ious Sec ion
o single-channel communica ions. The 4-by e con ol
commas a e eplaced by a combina ion o one da a by e plus a
3-by e con ol comma. The da a by e is hen used o encode
AER channel numbe .
IV. E
XPERIMENTAL
R
ESULTS
Fig. 4 shows a ypical se up whe e he p oposed
bidi ec ional se ial link has been used. The igu e shows an
AER e ina senso , connec ed by a pa allel AER connec o o
he AER-Node boa d [5]. The e ina communica es wi h he
Spa an6 FPGA on he AER-Node boa d, which
communica es h ough a SATA cable o one o he Spa an6
in a 48-chip SpiNNake boa d
1
[8]. The SpiNNake boa d
ecei es e en s, p ocesses hem and sends he esul ing e en
low back o he AER-Node boa d h ough he same SATA
wi e. The AER-Node boa d sends he esul s h ough ano he
pa allel AER po o one USBAERmini2 boa d [13] which
communica es h ough USB wi h a hos compu e o
display he esul s in eal ime.
Fig. 3: Block Diag am o Mul iple AER Channel Mul iplexing Scheme
Fig. 4: Example se up wi h DVS came a, a Spa an6 AER-Node PCB, a
48-chip SpiNNake PCB, and a USBAERmini2 compu e communica ion
PCB.
Fig. 5: Block Diag am o Expe imen al se up o es ing link pe o mance
be ween wo independen Spa an6 FPGA wi h sepa a e clock signal
oscilla o s in a 48-chip SpiNNake PCB boa d
1
The SpiNNake Boa d FPGA communica es wi h he SpiNNake chips h ough as 2-o -7 p edic i e
handshaking [11]. An imp o ed e sion o i is now pa o he o icial Spinn-Link elease [12].
Expe imen al e i ica ion and cha ac e iza ion o he p o-
posed communica ion scheme was pe o med using a pai o
Spa an6 FPGAs loca ed on a 48-chip SpiNNake PCB. Each
Spa an6 (XC6SLX45T-3) FPGA uses i s own 150MHz X al
oscilla o . Fo es ing he mul iplexed link a maximum
h oughpu , we used one GTP po o each FPGA. Each
FPGA uses i s local 75MHz e e ence clock. The se up was
con igu ed wi h 4 sepa a e bidi ec ional AER Channels o be
mul iplexed o e he SATA. Fo his, we used a pai o “Tes
Pa e n Gene a o ” (TPG) and “Tes Pa e n Checke ” (TPC)
inside FPGA and moni o hem using he ChipScope analyze
ool om Xilinx. The TPG p o ides a known sequence o
pa e ns, while he TPC checks and coun s e en e o s in his
sequence and compu es he e ec i e e en a e ecei ed
(excluding all con ol commas). In each FPGA, h ee
TPG/TPC pai s whe e clocked wi h he same clock han he
T anscei e /Mul iplexing co e discussed in Fig. 2 a n d Fig.
3. This is a clock a
=
75MHz de i ed om he ex e nal
150MHz X al e e ence oscilla o . The e o e, each o hese 3
synch onous TPGs can p o ide an e en a e o up o
75Meps (one pe clock cycle). The 4 h TPG/TPC pai was
clocked wi h an addi ional 67MHz o e i y he pe o mance
o he synch onize s. LVDS line a e was se a 3.0Gbps.
The se up was es ed du ing 65 hou s. None o he channels
did de ec a single e o in he ansmission. The link
bandwid h (75Meps) was sha ed by he ou Channels. The
TPGs clocked a 75MHz y o deli e da a a one e en pe
clock cycle, bu a e slowed down by he co esponding
encode in CH-MUX block i SATA link bandwid h is
eached. The o al bandwid h sha ed among he channels was
74.93Meps (99.90% o link bandwid h). This means ha
he emaining
0.10% was used by commas.
Ano he es was designed o s udy he e ec o clock
co ec ion comma inse ion pe iod on he link. In his es only
one channel wi h ull speed was used. Because e o s happen
in bu s s, e o a io in his case is no impo an . I is
impo an o know how long i will ake o lose alignmen . As
can be seen in Table 1, clock co ec ion inse ion pe iod wi h
less han 1.7ms (a equency o 75MHz, i will be one comma
pe 2
17
e en s) is enough o no expe ience e o s.
COMMA Pe iod Time o i s e o
In ini i e 138ms
56ms 138ms
28ms 196ms
14ms 247ms
7ms 340ms
3.5ms 325s
1.7ms In ini i e
V. A
CKNOWLEDGEMENTS
This wo k was suppo ed in pa by he Eu opean Resea ch
Council unde he Eu opean Union’s Se en h F amewo k
P og amme (FP/2007-2013)/ERC G an Ag eemen n.
320689, om he UK Enginee ing and Physical Sciences
Resea ch Council (EPSRC) unde G an EP/G015740/1, EU
FP7 g an 604102 “The Human B ain P ojec ” (HBP), EU
H2020 g an s 644096 “ECOMODE” and 687299
“NEURAM3”, b y Samsung Ad anced Ins i u e o
Technology g an NPP, by he Na ional Cen e o Resea ch
and De elopmen o Republic o Poland (NCBiR) unde g an
PBS3/A8/134/2015, by Spanish g an s om he Minis y o
Economy and Compe i i i y TEC2012-37868-C04-01
(BIOSENSE), TEC2015-63884- C2-1-P (COGNET), and
TEC2016-77785-P (COFNET) (wi h suppo om he
Eu opean Regional De elopmen Fund), and by
Andalusian g an TIC-6091 (NANO-NEURO). ARY is
suppo ed by a Spanish FPI Schola ship om he Minis y o
Economy and Compe i i i y.
R
EFERENCES
[1] P. O. Pouliquen and A. G. And eou, “Bi -Se ial Add ess-E en Rep-
esen a ion,” P oceedings o h e 33 d Annual Con e ence on
In o ma ion Sciences and Sys ems, Bal imo e MD, USA, Ma ch 1999.
[2] H. Be ge and P. Ha¨ lige , “High-Speed Se ial AER on FPGA,”
P
oc.
IEEE In . Symp. on Ci c. and Sys .
(ISCAS),
pp. 857–860, May 2007.
[3] D. B. Fasnach , e al., “A Se ial Communica ion In as uc u e o
Mul i-Chip Add ess E en Sys ems,” P oc. IEEE In . Symp. on Ci c.
and Sys .
(ISCAS),
pp. 648–651, May 2008.
[4] C.
Zama en˜
o-Ramos, e al., “Mul i-Cas ing Mesh AER: A Scalable
Assembly App oach o Recon igu able Neu omo phic S uc u ed AER
Sys ems. Applica ion o Con Ne s,” IEEE T ans. Biomedical Ci c. and
Sys ., ol. 7, no. 1, pp. 82–102, Feb. 2013.
[5] T. Iakymchuk, e al., “An AER Handshake-Less Modula In as uc u e
PCB wi h x8 2.5Gbps LVDS Se ial Links,” P oc. o he IEEE In .
Symp. on Ci cui s
and
Sys ems
,
pp. 1556–1559, 2014.
[6] L. Plana, J. Hea hco e, J. Peppe , S. Da idson, J. Ga side, S. Tem-
ple, and S. Fu be , “pI/O: A lib a y o FPGA designs and eusable
modules o I/O in SpiNNake sys ems,” A ailable on-line
a
:
h p://dx.doi.o g/10.5281/zenodo.51476, 2014.
[7] L. A. Plana, S. B. Fu be , S. Temple, M. Khan, Y. Shi, J. Wu, and
S. Yang, “A GALS In as uc u e o a Massi ely Pa allel Mul ip oces-
so ,” IEEE Design and Tes o Compu e s, pp. 454–463, Sep-Oc 2007.
[8] S. Fu be , e al., “O e iew o he SpiNNake Sys em A chi ec u e,”
IEEE T ans. Compu e s, ol. 62, no. 12, pp. 2454–2467, 2012.
[9] T. Se ano-Go a edona and B. Lina es-Ba anco, “A 128x128 1.5%
Con as Sensi i i y 0.9% FPN 3us La ency 4mW Asynch onous F ame-
F ee Dynamic Vision Senso Using T ansimpedance Ampli ie s,” IEEE
J.Solid-S a e Ci cui s, ol. 48, no. 3, pp. 827–838, 2013.
[10] P. A. F anaszek and A. X. Widme , “By e O ien ed DC Balanced
(0,4)
8b/10b Pa i ioned Block T ansmission Code,” US Pa en 4,486,738,
Dec. 4, 1984.
[11] A. R. Youse zadeh, e al., "Fas P edic i e Handshaking in Synch onous
FPGAs o Fully Asynch onous Mul i-Symbol Chip Links. Applica ion
o SpiNNake 2-o -7 Links," IEEE T ans. on Ci cui s and Sys ems, Pa
II, ol. 63, No. 8, pp. 763-767, Aug. 2016
[12] h ps://gi hub.com/SpiNNake Manches e /spio/blob/mas e /modules/
spinnake _link/spio_spinnake _link_ ecei e .
[13] R. Se ano-Go a edona, e al., “CAVIAR: A 45k Neu on, 5M Synapse,
12G Connec /s AER Ha dwa e Senso y-P ocessing-Lea ning-Ac ua ing
Sys em o High Speed Visual Objec Recogni ion and T acking,” IEEE
T ans. Neu al
Ne wo ks
,
ol. 20, no. 9, pp. 1417–1438, Sep . 2009.
Table 1 Expe imen ally Measu ed ime o i s e o by sweeping he
clock co ec ion comma pe iod in TX block