VLSI implementation of a fully parallel stochastic neural network

i VLSI Implemen a ion o a Fully Pa allel S ochas ic Neu al Ne wo k J.M.Que o, J.G.O ega, C.L.Jane and L.G.F anquelo IEEE h EMBER Dp o. de Ingenie ia de Sis emas y Au omii ica Uni . de Se illa, Spain Abs ac - In his pape we p esen a pu ely digi al s ochas ic implemen a ion o mul ilaye neu al ne wo ks. We ha e de elopped his implemen a ion using an a chi ec u e ha pe mi s he addi ion o a e y la ge numbe o synap ic connec ions, p o ided ha he neu on’s ans e unc ion is he ha d limi ing unc ion. The exp ession ha ela es he design pa ame e , ha is, he maximun pulse densi y, wi h he accu acy o he ope a ions has been used as design c i e ium. The esul ing ci cui is easily con igu able and expandable. I. INTR.ODUCTION S ochas ic neu al ne wo ks a chi ec mii es ha e ecen ly ecei ed much a en ion [1],[2]. The use o hese a chi ec u es dec eases he amoun o ha dwa e needed o ea?ize he ope a ions in ol ed in neu ons. I we look a he ans e unc ion o a neu on (3; = a(Cj;uijyj Ii)) we ealize ha we ha e o e alua e a se o a i hme ic ope a ion, mo e p ecisely, a se o p oduc s and addi ions. In o de o make e icien elec onic implemen a ions o neu al ne wo ks con aining a high numbe o neu ons, i woiild be desi a.ble ha, hese ope a ions could be pe o med by simple ci cui y. S ochas ic logic sys ems ealize pseudoanalog ope a ions using s ochas ically coded pulse sequences. These pulse sequences ca,n be gene a ed using a g oup o digi al compa a o s. A digi al codi ica ions o a iables a e compa ed wi h unco ela ed andom numbe s p oducing unco ela ed s ochas ic signals whose alues andomly ake alues 0 o 1. The a e age o hese alues can be < 1, wi h Rmax iewed as an analog alue in he ange [-a,a] o [O,o], whe e a = andomma= - he maximum alue ha ca,n be s o ed in R; and ~undom~,, is he maximum andom numbe gene a ed. Rmax Mul iplica ion o wo s ochx ic pulse sequences should p oduce ano he s ochas ic s eam o pulses whose i ing p obabili is he p oduc o he inpu i ing p obabili ies. This can be easily achie ed i he inpu sequences a e s ochas ically independen . The ci cui ha implemen s his ope a ion is a simple AND ga e. 0-7803-1901-x194 $4.00 01994 IEEE 2040 S ochas ic summa ion is a much mo e di icul ope a ion o pe o m, specially i he e ms o be added a e signed. Two ypes o ci cui s ha e been desc ibed in he bibliog aphy. One is he OR ga e and he o he is he up-down coun e . The up-down coun e s echnique, al hough is widely used when implemen ing ’Hop ield’s neu al ne wo k, has a e y impo an d awback. Pulses coming om o he neu ons ha e o be mul iplexed in ime (i.e. se zdized) leading o high compu a ion, i” i he ne wo k has many neu ons and many connec ions pe neu on. I should be also poin ed ou ha he se ializa ion ha akes place is only e icien in neu d ne wo ks whose neu ons a e ully connec ed . I wo pulse sequences am e ed o an OR ga e he ou pu i ing p obabili y would be equd o he sum o bo h i ing p obabili ies i he pulse sequences o be added did no o e lap. This OR-based add unc ion is hus dis o ed by pulse o e lap. I only posi i e e ms a e o be added pulse o e lap is no a d awback because i p oduces he non linea ans e unc ion o a neu on. I signed inpu s a e o be added linea beha io is an essen ial ea u e. Howe e his app oach is ex emely in e eshing due o he simplici y o he equi ed ha dwa e which would pe mi he digi al implemen a ion o a bi a y neu a.1 ne wo ks con aining a e y high numbe o neu ons. 11. PROPOSED STOCHASTIC: ARCHITECTURE I has been p o ed ha he OR. ga e based summa ion is ex emely e icien i we es ic ou sel es o neu al ne wo ks in which neu ons ake only wo disc e e alues. Due o he ac ha he exponen ial Iiinc, ion is iiioiio oiiicl lly inc easing, acid akin in o accoun i s p ope ies, i can be deduced ha sign(Cizy xi) = sign(niz;”- e-”: - l-Iizy e-“i ), whe e supe sc ip s + and - a e ex ended o posi i e and nega i e e ms espec i ely. 5 . The las exp ession can be ega. ded as he compa. ison o wo pulse s eams gene a ed by wo s ochas ic mul iplie s, he e o e no adde is needed. The only p oblem is o e alua e he exponen ial ans o ma ion in a. e icien way. I he neu al ne wo k has been adimen ionalized in such a way ha all e ms o be agg ega ed ake alues anging om ze o o a small numbe a close enough o ze o, e-”% can be app oxima ed by 1 - 2;. The esul ing s uc u e is shown in Fig.1. Inpu pulses a e in e ed by no ga es and hen ed o he co esponding AND ga es by he i s se o logic blocks. I should be poin ed ou ha hese logic blocks should main ain i s ou pu signal a high le el while no being d i en by he inpu pulses. The ou pu logic Mock e alua es he neu on ou pu . I only he “posi i e” pulse is a high le el he coun e is inc emeii ed by one and i he pulse is “nega i e” he coun e is hen deuemen ed by one. The sign bi is chmged i a ze o c ossing akes place. I a second o de app oxima. ion o he exponen id , a.ns o ma. ,ion was made, we could la gely inc ease he alue o U a he expenses o ci cui ’s complexi y (see Fig. 2). Due o he ac ha, he exponen ial ans o ma ion is no e alua ed exac ly, i ollows ha he addi ion opem ion has a. limi ed accu acy. I he o d neu on’s exci a o y inpu and he o al 2041 i ‘I neu on’s inhibi o y inpu a e e y closed numbe s, he neu on’s ou pu may be unco ec . I has been p o ed ha he accu acy o he addi ion is bounded by he ollowing exp ession: whe e 2 is he a e e o o he addi ion ope a ion. I he mos accu a e compa ison ha has o be e alua ed is a known quan i y, i ollows ha a has o be chosen so ha < w, whe e DIFF is he di e ence be ween he o al posi i e neu on’s inpu and he o al nega i e neu on’s inpu . I he ne inpu s can ake any alues, he limi ed accu acy o he addi ion de e mines a egion whe e he neu on’s ou pu will be unce ain. 111. IMPLEMENTATION In his sec ion se e al design issues, such us andom numbe gene a ion, ne wo k con igu a ion, ne wo k expansion a.nd ha dwa e implemen a ion, a e desc ibed. a) R.a.ndom numbe gene a. ion Linea Feedback Shi R.egis e (LFSR) is one o he mos s udied digi al echniques o gen- e a e pseudo- andom numbe s [4]. Howe e , all he esul s gi en in p e ious sec ions a e alid i pulse secuences a e pu ely andom -specially hose in ol ed in he’same neu on. In ou design we ha e used a 9-bi LFSR. wi h do = XOR(q0, qd). We can ob ain he c oss-co ela ion (C ) be ween shi ed sequences n(k) and n(k - p), whe e n,(k) s ands o he o iginal no malized sequence and n(k + p) is i s p-s eep ahead shi sequence. Fig 3 ep esen s he c oss-co ela ion be ween shi ed mdom numbe sequences. I is clea ha we need a, 8-s ep shi a leas o gene a e pulse se- quences as unco ela ed as possible. These shi s ha e been macle using 8-XOR ga es. The numbe o unco ela ed pseudo- andom sequences ha can be simul aneously gene a ed is 2’/8 N 64. This numbe can be inc eased using a la ge LFSR. b) Ne wo k con igu a. ion We ha e iised pipeliiicd pa allel egis e s o allow neu al ne wo k p og ammabili y wi h a minimun numbe o inpu s. The size o hese egis e s is ela ed o he size o he LFSR and he maximum ole a ed e o in he exponen ial app oxima ion. In his case we ha e choosen a 7-bi egis e o weigh s o age. I we conside inpu sequences o he ne wo k wi h a maximun p obabili y o 0.5, he esul ing niul ipli a ioii has a mean o $0.5 = 0.125. Acco ding o Fig. 4, i leads o an exponen ial app oxima, ion e o o 6% (linea ampp oxima ion o he exponen ial ans o ma ion). c) Ne wo k expansion One o he mos ema, kable ea u es o he p oposed aachi ec u e is ha he numbe o 2042 connec ions can be easily inc eased. Addi ions a e caIcula, ed like p oduc s, so ha se s o weigh s can be added using !P+ and q- signals wi hou addi iona.1 ha dwa e (see Fig. 1). The numbe o hidden laye s ca,n also he inc eased wi h addi ional ci cui s in cascade. (1) Hwdwa e implemen a ion The p oposed s ochas ic a chi ec u e has been implemen ed using EDGE wi h ES2 1.5~ S anda Cell lib a ies. The esul ing ci cui con ains wo laye s wi h i e inpu unsigned pulse signals, i e hidden neu ons and one ou pu neu on. Ou pu neu on Q+ and Q- signals a e accessible, so ha. he numbe o hidden neu ons ma,y be inc eased. IV. RESULTS In o de o es he beha iou o he I.C., we ha e de elopped he con olled p esen ed in [5]. This applica ion consis s on a pe cep on ha app oxima es a classi ica ion su ace desc ibed by a 196-poin a ay. .4 wo laye pe cep on wi h 10 hidden neu ons ha e been con igu ed using Back p opaga ion. Two 1.C.s ha e been used o implemen his applica, ion. All aining inpu pa e ns a e co ec ly classi ied by he ne wo k. The ansien esponse o he ou pu neu on o he inpu ec o (0.0867,0.42,0.39) is shown in Fig. 5, whe e he clock a e is 7.5MHz. Fig. 6 shows ,he beha io o he ne wo k as he inpu ec o c osses he decision su ace. In his igu e he desi ed ne wo k’s esponse and he ac ual ne wo k’s esponse ha e been ep esen ed. Se e al easons explain he sligh , di e ences be ween he wo g a,phics: he disc e iza ion o he synap ic weigh s, he limi ed ac,cu a.c.y o he s ochas ic a,ddi ions and he use o pseudo andom sequences ins ead o eal andom numbe s. V. CONCLUSIONS In his pa.pe we ha e p esen ed a ha. wa, e implemen a ion o a. mul ilaye neu al ne wo k using he s ochas ic a. chi ec u e ha was p oposed in [3]. This a chi ec u e can be easily expanded, and he ci cui has been designed so ha, any niul ila,ye ne wo k can be implemen ed by adding an app op ia e numbe o 1.C.s. As a.n example we ha e implemen ed he con olle wi h pe cep on- like s uc u e which was p esen ed in [5]. The esponse o he ne wo k ma ches wi h a high deg ee o accu acy he heo e ical con olle ’s esponse. Re e ences [l] Y. Kon lo and Y. Sawa. la. Func ional Ahili ies o a S ochas ic Logic Neu al Ne wo ks IEEE T ans. on Nein~nl Ne wo ks, o1.3, pp.434-443, 1992. 2043 [2] D.E. Van den Bou and T.K. Mille 111. A Digi al A chi ec u e Employing S ochas icism o he Simula ion o Hop ield Neu al Ne s. IEEE T ans. on Ci cui and Sys ems, o1.36, pp. 732-738. 1989. [3] C.L. Jane , J.M. Que o and L.G. F anquelo. Fully Pa allel Summa ion in a New S ochas ic Neu al Ne wo k A chi ec u e. IEEE In . Con . on Neu al Ne wo ks, San F ancisco, pp. 1498- 1503. 1993. [4] W. Pe e son. E o Co ec ing Codes. MIT P ess. 1992. [5] .J.M. Que o, J.M. Ca. a.sco and L.C. F a.nqnelo. Ada.p a i e Ene gy Feed-Back Con ol o Resonan Con e e s Using Neu al Ne wo ks. 23 d Annual Powe Elec onics Specialis s Con- e ence, Toledo, Spain, pp. 800-806. 1992. Mpxiniun puke mi y: a (a<cl Figu e 1: Neu on's s uc u e Figu e 2: E alua. ion c.i cui o he second o de app oxima ion o he exponen ial ans o ma ion. 2044 0.045 0.0. 0.035 0.03 Y.UI5 0.0) 0.015 0.01 0.005 0 -0 005 1 1- Figu e 3: C oss-co ela ion be ween p-s ep ahead shi pseudo- andom numbe sequences gene a ed by a 9-bi LFSR) 01 0.05 I -O."! Figu e 4: No malized exponen ial a.ns o ma.- ioii e o , o he i s , second and hi d o dc app oxima ion. DO Figu e 5: T ansien esponse o he ou pu neu- on o he inpu ec o (0.0867,0.42,0.39) Figu e 6: Ne wo k's desi ed and ac ual esponse as he inpu ec o c osses he decision su ace. 2045