Multiplexing AER Asynchronous Channels over LVDS Links with Flow-Control and Clock-Correction for Scalable Neuromorphic Systems

Mul iplexing AER Asynch onous Channels o e LVDS Links wi h Flow-Con ol and Clock-Co ec ion o Scalable Neu omo phic Sys ems A. Youse zadeh 1 , M. JabáoĔski 2 , T. Iakymchuk 3 , A. Lina es-Ba anco 4 , A. Rosado 3 , L. A. Plana 5 , T. Se ano-Go a edona 1 , S. Fu be 5 , and B. Lina es-Ba anco 1 1 Ins i u o de Mic oelec ónica de Se illa (CSIC and Uni . de Se illa), Se illa, Spain { eza, be nabe}@imse-cnm.csic.es 2 AGH Uni e si y o Science and Technology, K akow, Poland 3 School o Enginee ing (Uni . o Valencia), Spain 4 Dep . o Compu e A chi ec u es (Uni . o Se illa), Spain 5 Dep . Comp. Science, Uni e si y o Manches e , UK Abs ac — Add ess-E en -Rep esen a ion (AER) is a widely ex ended asynch onous echnique o in e changing “neu al spikes” among di e en ha dwa e elemen s in Neu omo phic Sys ems. Con en ional AER links use pa allel physical wi es oge he wi h a pai o handshaking signals (Reques and Acknowledge). He e we p esen a ully se ial implemen a ion using bidi ec ional SATA connec o s wi h a pai o LVDS (low ol age di e en ial signaling) wi es o each di ec ion. The p oposed implemen a ion can mul iplex a numbe o con en ional pa allel AER links pe LVDS physical connec ion. I uses low con ol, clock co ec ion, and by e alignmen ech niques o ansmi 32-bi add ess e en s eliably o e mul iplexed se ial connec ions. The se up has been es ed using comme cial Spa an6 FPGAs eaching a maximum e en ansmission speed o 75Meps (Mega E en s pe second) o 32-bi e en s a 3.0Gbps line da a a e. Keywo ds— Neu omo phic Sys ems, Vi ual Wi ing, AER (Add ess E en Rep esen a ion), Scalable Neu omo phic Sys ems. I. I NTRODUCTION Add ess E en Rep esen a ion (AER) is a popula “ i ual wi ing” echnique used by many neu omo phic ha dwa e enginee s o in e connec spiking neu omo phic sys ems. As neu omo phic sys ems ha e been scaling up in size and complexi y o e he yea s, esea che s ha e de eloped mo e complex and sma e AER “ a ia ions” o imp o e e iciency, econ igu abili y and eliabili y. Since e y ea ly [1] i became appa en ha he o iginal pa allel-AER (pAER) bus bulkiness would se iously limi he scalabili y o AER sys ems o a bi a y size, and esea che s s a ed o look a se ial connec i i y op ions. Fully bi -se ial Low-Vol age- Di e en ial-Signaling (LVDS) allow o mul i-gigabi -pe - second communica ions wi h only a pai o wi es. Howe e , wi h only wo uni-di ec ional di e en ial wi es i is no ob ious o implemen a handshaking p o ocol pe e en ansmission, no a low-con ol scheme o signal da a conges ion a he ecei e side. Be ge e al. [2] expe imen ed wi h he 2.5Gbps (Giga-bi -pe -second) LVDS IP block a ailable in he Vi exII-P o Xilinx FPGAs, achie ing 41.66Meps (Mega-e en s-pe -second) o 16-bi AEs (Add ess E en s). Howe e , he e was no hand-shaking no low-con ol mechanism o a oid da a loss in case he ecei e side would be empo a ily slowe han he ansmi e side. Fasnach e al. [3] epo ed a bi -se ial in e ace based on o - he-shel 16-bi Se ialize /Dese ialize comme cial componen s (TLK 2501/3101) connec ed o a Spa an3E pa allel e en AER p ocesso , using a second backwa ds LVDS link o low con ol signaling. The bi -se ial link could ope a e a 2.5Gbps (TLK 2501) o 3.125Gbps (TLK3101) line speed and also used 8b/10b encoding o allow o idle commas. On he LVDS backwa d link he ecei e pu s a squa e wa e whose equency signals whe he o s op o esume e en ansmission om he sende . Using his se up he link could ansmi 32-bi e en s a a maximum a e o 62.5Meps ( o 2.5Gbps) o 78.125Meps ( o 3.125Gbps), a he cos o sac i icing one LVDS backwa d link o low con ol. Zama en˜ o e al. [4] de eloped a bi-di ec ional LVDS link using Vi ex6 FPGA Rocke -I/O IPs. In his scheme wo 2.5Gbps LVDS links a e used, each o communica ing 32-bi e en s in each di ec ion wi h 8b/10b encoding. Flow-con ol in each di ec ion is implemen ed by special con ol symbols Fig. 1: Example se up o se en een Spa an6 PCBs communica ing h ough bidi ec ional LVDS SATA o hogonal links, plus a DVS came as and USB-AER boa d communica ing h ough pa allel AER buses wi h one o he Spa an6 PCBs. Fig. 2: Block Diag am o Bidi ec ional LVDS link communica ing wo pa allel AER asynch onous links o opposi e di ec ion sen in he opposi e link di ec ion o oggle be ween he s op and esume s a es. Howe e , his design did no include any clock-co ec ion suppo , which limi s he app oach o he case when all FPGAs use he same physical e e ence clock, hus hampe ing scalabili y. All p e ious asynch onous pAER o bi -se ial AER con e sion schemes ope a e co ec ly i he e is only one clock domain. Howe e , his is no iable when one wan s o in e connec mul iple FPGAs, each wi h i s own local-clock synch onous e en p ocessing subsys em, oge he wi h mul iple bidi ec ional LVDS links pe FPGA. In his case, each FPGA will ha e one o mo e local clock domains, which can be in e e ing wi h he clock domains o neighbo ing FPGAs. Unde hese ci cums ances, i is necessa y o use some clock co ec ion echnique in o de o compensa e o clock equency/phase d i s and a oid sudden by e misalignmen p oblems and da a loss. In a p elimina y wo k [5] we exploi ed he use o elas ic bu e s a ailable wi hin Xilinx Rocke -I/O se ial LVDS IPs, al hough i was used o ully synch onous handshake-less sys ems deployed o e mul iple FPGAs. Each link was unidi ec ional since he backwa d LVDS pa h was ully used o handling low con ol. Mo e ecen ly, he SpiNNake eam has de eloped bidi ec ional bi -se ial LVDS links [ 6 ] o bundle eigh 2- o -7 AER mul i-symbol in e -SpiNNake -chip links [7] in o one bi -se ial SATA link. This scheme has been de eloped o in e connec mul iple (up o 1200) 48-chip SpiNNake Boa ds [8]. The scheme uses low-con ol, clock co ec ion, oge he wi h a complex aming p o ocol ha samples he eigh channels and pe o ms CRC (Cyclic Redundancy Checks) o inc ease eliabili y. This in oduces ex a o e heads, limi ing he maximum h oughpu heo e ically o 50Meps (maximum 6.25Meps pe channel). He e we p esen an ex ended e sion o he one epo ed ea lie [5], bu wi h ully bidi ec ional bi -se ial LVDS communica ion capabili y, wi h oken-based low-con ol p o ocol oge he wi h clock co ec ion capabili y, as well as a obus in e ace o con en ional pa allel AER po s (like hose used in AER senso chips) wi h 4-phase handshaking asynch onous communica ion. Using a 3.0Gbps LVDS line ansmission a e i is possible o achie e 32-bi 75.0Meps sus ained ansmission in each di ec ion o he link. Fig. 1 shows an example a ge se up consis ing o an a ay o 17 AER-Node Boa ds [ 5 ] in e connec ed h ough SATA o hei neighbo s. Addi ionally, an AER e ina senso and a USB- AER boa d [13] a e connec ed o he se up h ough pa allel AER buses [9]. II. B IDIRECTIONAL LVDS AER B IT -S ERIAL L INK WITH F LOW -C ONTROL , C LOCK C ORRECTION AND B YTE -A LIGNMENT 8b/10b encoding [10] ans o ms 8-bi by es in o 10-bi wo ds while dc balancing ze os and ones. Ex a dc-balanced 10-bi wo ds a e a ailable, which can be used by he use as “command” cha ac e s, also called “k-cha s” o “commas”. He e we use some o hese k-cha s o pe o m low con ol, by e alignmen s and clock co ec ions. Fig. 2 shows a block diag am o a bidi ec ional AER link using wo FPGA PCBs connec ed h ough a single SATA cable (con aining wo pai s o wi es). Each FPGA connec s o one hand-shaken 32-bi pAER sende and one pAER ecei e . The “w appe ” block con ains a w appe ansmi e sub-block wTX and a w appe ecei e sub-block wRX. wTX akes as inpu a 32-bi clock-synch onous “DATA” wo d and a 4-bi “k-cha ” wo d. Each o he ou “k-cha ” bi s indica e whe he he co esponding ou by es in he 32-bi “DATA” wo d a e ei he a con ol comma by e o a egula use da a by e. In ou case, LVDS line a es we e 3.0Gbps, ob ained om a low ji e di e en ial 150MHz e e ence X al oscilla o on he FPGA PCB. This means ha he 32-bi e en da a, ans o med in o a 40-bi sequence by 8b/10b encoding, needs 40 / (3 × 10 9 Hz) = 13.3ns o be ansmi ed. The w appe p o ides wo e e ence clocks o he use ci cui y. One a equency = 1 / 13 . 3 ns = 75.0MHz, a he ising edges o which he use ci cui y has o p o ide he 32-bi pa allel “DATA” wo d and he co esponding 4-bi “k-cha ” wo d. The o he e e ence clock has equency 4 = 300MHz (because he 32-bi “DATA” wo d con ains 4 by es) and is synch onized o clock .La ency o ansmission in ou se up is 20 clock cycles (266ns). The use designed ci cui y wi hin each FPGA in Fig. 2 comp ises 3 blocks: 1) T ansmi e Block (TX). This block handles he asynch onous handshaking wi h he inpu 32-bi AERin po , he synch oniza ion be ween he asynch onous and synch onous domains and “S op/Run” s a e o he ansmission o da a which is used by he low con ol p o ocol. Also, his block p o ides o each clock cycle he 32- bi “DATA” wo d and he 4-bi “k-cha ” wo d equi ed by he “w appe ” (wTX), gene a es s a up by e alignmen sequences, and inse s con ol symbols o low con ol, clock co ec ion, idle commas, and pe iodic commas o alignmen . 2) Recei e Block (RX). This block ecei es and sepa a es da a and con ol commas. Da a symbols a e sen o he FIFO block, while con ol commas a e in e p e ed and execu ed o p ope low con ol, wo d ( e)alignmen , and clock co ec ion. 3) FIFO block. This block accumula es 32-bi synch onous da a symbols om he RX block in o a FIFO egis e , while i emp ies he FIFO by sending da a ou o he AERo po . This block handles he asynch onous handshaking wi h he ou pu 32-bi AERo po and he synch oniza ion wi h he synch onous da a “DATA” clock domain. I he FIFO ge s illed up abo e a h eshold i will igge he low con ol mechanism. Fig. 2 illus a es low con ol mechanism. I FIFO block is close o ge ing ull, i sends a 4-by e s op oken con ol comma h ough he backwa d link o i s co esponding TX block on he ansmi e side, se ing i in o ‘s op’ mode. In his mode Ack will no be acknowledged and he pa allel AER po s ops aking new e en s. Once he FIFO block has enough ee space, a 4-by e esume oken is sen in he same manne o disable TX block s op mode and esume communica ion. Fo clock co ec ion, pe iodic 4-by e con ol commas a e inse ed in he da a low by he ansmi e . I he ecei e clock is as e /slowe i will e ase/inse such commas h ough i s elas ic bu e , hus equalizing he e ec i e da a a e be ween ansmi e and ecei e side. Addi ionally, when no da a a e sen , 4-by e idle commas a e inse ed o keep he LVDS link unning and aligned. III. M ULTIPLE AER C HANNELS M ULTIPLEXING Fo mul iple AER channels mul iplexing he app oach shown in Fig. 3 is used. The op q bi s o he 32-bi da a wo d a e used o encode k AER channels (2 q  k). Each AER channel TX-FIFO eques s access o an encode in he CH-MUX block. Once access is g an ed, communica ion goes h ough he CH- MUX in a simila way as was desc ibed in he p e ious Sec ion o single-channel communica ions. The 4-by e con ol commas a e eplaced by a combina ion o one da a by e plus a 3-by e con ol comma. The da a by e is hen used o encode AER channel numbe . IV. E XPERIMENTAL R ESULTS Fig. 4 shows a ypical se up whe e he p oposed bidi ec ional se ial link has been used. The igu e shows an AER e ina senso , connec ed by a pa allel AER connec o o he AER-Node boa d [5]. The e ina communica es wi h he Spa an6 FPGA on he AER-Node boa d, which communica es h ough a SATA cable o one o he Spa an6 in a 48-chip SpiNNake boa d 1 [8]. The SpiNNake boa d ecei es e en s, p ocesses hem and sends he esul ing e en low back o he AER-Node boa d h ough he same SATA wi e. The AER-Node boa d sends he esul s h ough ano he pa allel AER po o one USBAERmini2 boa d [13] which communica es h ough USB wi h a hos compu e o display he esul s in eal ime. Fig. 3: Block Diag am o Mul iple AER Channel Mul iplexing Scheme Fig. 4: Example se up wi h DVS came a, a Spa an6 AER-Node PCB, a 48-chip SpiNNake PCB, and a USBAERmini2 compu e communica ion PCB. Fig. 5: Block Diag am o Expe imen al se up o es ing link pe o mance be ween wo independen Spa an6 FPGA wi h sepa a e clock signal oscilla o s in a 48-chip SpiNNake PCB boa d 1 The SpiNNake Boa d FPGA communica es wi h he SpiNNake chips h ough as 2-o -7 p edic i e handshaking [11]. An imp o ed e sion o i is now pa o he o icial Spinn-Link elease [12]. Expe imen al e i ica ion and cha ac e iza ion o he p o- posed communica ion scheme was pe o med using a pai o Spa an6 FPGAs loca ed on a 48-chip SpiNNake PCB. Each Spa an6 (XC6SLX45T-3) FPGA uses i s own 150MHz X al oscilla o . Fo es ing he mul iplexed link a maximum h oughpu , we used one GTP po o each FPGA. Each FPGA uses i s local 75MHz e e ence clock. The se up was con igu ed wi h 4 sepa a e bidi ec ional AER Channels o be mul iplexed o e he SATA. Fo his, we used a pai o “Tes Pa e n Gene a o ” (TPG) and “Tes Pa e n Checke ” (TPC) inside FPGA and moni o hem using he ChipScope analyze ool om Xilinx. The TPG p o ides a known sequence o pa e ns, while he TPC checks and coun s e en e o s in his sequence and compu es he e ec i e e en a e ecei ed (excluding all con ol commas). In each FPGA, h ee TPG/TPC pai s whe e clocked wi h he same clock han he T anscei e /Mul iplexing co e discussed in Fig. 2 a n d Fig. 3. This is a clock a = 75MHz de i ed om he ex e nal 150MHz X al e e ence oscilla o . The e o e, each o hese 3 synch onous TPGs can p o ide an e en a e o up o 75Meps (one pe clock cycle). The 4 h TPG/TPC pai was clocked wi h an addi ional 67MHz o e i y he pe o mance o he synch onize s. LVDS line a e was se a 3.0Gbps. The se up was es ed du ing 65 hou s. None o he channels did de ec a single e o in he ansmission. The link bandwid h (75Meps) was sha ed by he ou Channels. The TPGs clocked a 75MHz y o deli e da a a one e en pe clock cycle, bu a e slowed down by he co esponding encode in CH-MUX block i SATA link bandwid h is eached. The o al bandwid h sha ed among he channels was 74.93Meps (99.90% o link bandwid h). This means ha he emaining 0.10% was used by commas. Ano he es was designed o s udy he e ec o clock co ec ion comma inse ion pe iod on he link. In his es only one channel wi h ull speed was used. Because e o s happen in bu s s, e o a io in his case is no impo an . I is impo an o know how long i will ake o lose alignmen . As can be seen in Table 1, clock co ec ion inse ion pe iod wi h less han 1.7ms (a equency o 75MHz, i will be one comma pe 2 17 e en s) is enough o no expe ience e o s. COMMA Pe iod Time o i s e o In ini i e 138ms 56ms 138ms 28ms 196ms 14ms 247ms 7ms 340ms 3.5ms 325s 1.7ms In ini i e V. A CKNOWLEDGEMENTS This wo k was suppo ed in pa by he Eu opean Resea ch Council unde he Eu opean Union’s Se en h F amewo k P og amme (FP/2007-2013)/ERC G an Ag eemen n. 320689, om he UK Enginee ing and Physical Sciences Resea ch Council (EPSRC) unde G an EP/G015740/1, EU FP7 g an 604102 “The Human B ain P ojec ” (HBP), EU H2020 g an s 644096 “ECOMODE” and 687299 “NEURAM3”, b y Samsung Ad anced Ins i u e o Technology g an NPP, by he Na ional Cen e o Resea ch and De elopmen o Republic o Poland (NCBiR) unde g an PBS3/A8/134/2015, by Spanish g an s om he Minis y o Economy and Compe i i i y TEC2012-37868-C04-01 (BIOSENSE), TEC2015-63884- C2-1-P (COGNET), and TEC2016-77785-P (COFNET) (wi h suppo om he Eu opean Regional De elopmen Fund), and by Andalusian g an TIC-6091 (NANO-NEURO). ARY is suppo ed by a Spanish FPI Schola ship om he Minis y o Economy and Compe i i i y. R EFERENCES [1] P. O. Pouliquen and A. G. And eou, “Bi -Se ial Add ess-E en Rep- esen a ion,” P oceedings o h e 33 d Annual Con e ence on In o ma ion Sciences and Sys ems, Bal imo e MD, USA, Ma ch 1999. [2] H. Be ge and P. Ha¨ lige , “High-Speed Se ial AER on FPGA,” P oc. IEEE In . Symp. on Ci c. and Sys . (ISCAS), pp. 857–860, May 2007. [3] D. B. Fasnach , e al., “A Se ial Communica ion In as uc u e o Mul i-Chip Add ess E en Sys ems,” P oc. IEEE In . Symp. on Ci c. and Sys . (ISCAS), pp. 648–651, May 2008. [4] C. Zama en˜ o-Ramos, e al., “Mul i-Cas ing Mesh AER: A Scalable Assembly App oach o Recon igu able Neu omo phic S uc u ed AER Sys ems. Applica ion o Con Ne s,” IEEE T ans. Biomedical Ci c. and Sys ., ol. 7, no. 1, pp. 82–102, Feb. 2013. [5] T. Iakymchuk, e al., “An AER Handshake-Less Modula In as uc u e PCB wi h x8 2.5Gbps LVDS Se ial Links,” P oc. o he IEEE In . Symp. on Ci cui s and Sys ems , pp. 1556–1559, 2014. [6] L. Plana, J. Hea hco e, J. Peppe , S. Da idson, J. Ga side, S. Tem- ple, and S. Fu be , “pI/O: A lib a y o FPGA designs and eusable modules o I/O in SpiNNake sys ems,” A ailable on-line a : h p://dx.doi.o g/10.5281/zenodo.51476, 2014. [7] L. A. Plana, S. B. Fu be , S. Temple, M. Khan, Y. Shi, J. Wu, and S. Yang, “A GALS In as uc u e o a Massi ely Pa allel Mul ip oces- so ,” IEEE Design and Tes o Compu e s, pp. 454–463, Sep-Oc 2007. [8] S. Fu be , e al., “O e iew o he SpiNNake Sys em A chi ec u e,” IEEE T ans. Compu e s, ol. 62, no. 12, pp. 2454–2467, 2012. [9] T. Se ano-Go a edona and B. Lina es-Ba anco, “A 128x128 1.5% Con as Sensi i i y 0.9% FPN 3us La ency 4mW Asynch onous F ame- F ee Dynamic Vision Senso Using T ansimpedance Ampli ie s,” IEEE J.Solid-S a e Ci cui s, ol. 48, no. 3, pp. 827–838, 2013. [10] P. A. F anaszek and A. X. Widme , “By e O ien ed DC Balanced (0,4) 8b/10b Pa i ioned Block T ansmission Code,” US Pa en 4,486,738, Dec. 4, 1984. [11] A. R. Youse zadeh, e al., "Fas P edic i e Handshaking in Synch onous FPGAs o Fully Asynch onous Mul i-Symbol Chip Links. Applica ion o SpiNNake 2-o -7 Links," IEEE T ans. on Ci cui s and Sys ems, Pa II, ol. 63, No. 8, pp. 763-767, Aug. 2016 [12] h ps://gi hub.com/SpiNNake Manches e /spio/blob/mas e /modules/ spinnake _link/spio_spinnake _link_ ecei e . [13] R. Se ano-Go a edona, e al., “CAVIAR: A 45k Neu on, 5M Synapse, 12G Connec /s AER Ha dwa e Senso y-P ocessing-Lea ning-Ac ua ing Sys em o High Speed Visual Objec Recogni ion and T acking,” IEEE T ans. Neu al Ne wo ks , ol. 20, no. 9, pp. 1417–1438, Sep . 2009. Table 1 Expe imen ally Measu ed ime o i s e o by sweeping he clock co ec ion comma pe iod in TX block