A trace-scaling agent for parallel application tracing.

A T ace-Scaling Agen o Pa allel Applica ion T acingJ Felix F ei ag. Jo di Caube , Jesus Laba a Compu e A chi ec u e Depa men (DAC) Eu opean Cen e o Pa allelism o Ba celona (CEPBA) Uni e .'Ji a Poli ecnica de Ca alunya (UPC) { elix,jo dics,jesus}@ac. upc.es Abs ac equi emen o ace iles. We show ha he agen can ob ain such an unde s anding au oma ically a un ime wi hou p og amme in e en ion o suppo . The emainde o he pape is s uc u ed as ollows: In sec ion 2 we desc ibe scalabili y p oblems o acing mechanisms. Sec ion 3 shows he implemen a ion o he ace-scaling agen . Sec ion 4 desc ibes some applica ions and esul s o scaled acing. Sec ion 5 con ains u he disussion o ou app oach. In sec ion 6 we conclude he pape . T acing and pe o mance analysis ools a e an impo an componen in he de elopmen o high pe o mance applica ions. T acing pa al'el p qg ams wi h cu en acing ools. howe e . easily leads o la ge ace iles wi h hund eds o lUegaby es. The s~o age. isualiza ion, and analysis o such ace iles is o en di icul . We p opose a ace-scaling agen o acing p~ al'el applica ions. which lea ns he applica ion behaVio in un ime and achie es a small. easy o handle ade. The agen dynamically iden i ies he amoun o in o b a ion needed o cap u e he applica ion beha io . This knowledge acqui ed a un ime allows eco ding o~ly he non-i e a i e ace i i o ma ion, which d as ical'y !duces he size o he ace ile. 2. Scalabili y o acing mechanisms 2.1. P oblems associa ed o la ge aces The pe onnance analysis o pa allel p og ams easily leads o a la ge numbe o ace iles, since o en se e al execu ions o he ins umen ed applica ion a e ca ied ou in o de o obse e he applica ion beha io unde sligh ly changed condi ions. Ano he eason why se e al aces a e needed is o s udy how he applica ion scales. All hese aces o he di e en con igu a ions o he applica ion and he en i onmen (numbe o p ocesso s, algo i hmic changes, ha dwa e coun e s, ...) equi e s o age space. Visualiza ion packages ha e di icul ies in showing such la ge aces e ec i ely. La ge aces make he na iga ion (zooming, o wa d/backwa d anima ion) h ough hem e y slow and equi e he machine whe e he isualiza ion package is un o ha e a la ge physical memo y. O he n'ise, he esponse ime o he ool inc eases signi ican ly, s ongly a ec ing he mo i a ion o he p og amme o ca y ou he pe onnance analysis. The high amoun o edundan ace in onna ion in la ge ace iles hides he ele an de ails o he applica ion beha io . When isualizing such la ge aces, zooming down o iden i y he applica ion s uc u e becomes an ine icien ask o he p og am analys . O en, he analys needs o ha e a ce ain unde s anding o he applica ion in o de o ca y ou an e icien pe onnance analysis. I. In oduc ion Pe o mance analysis ools a e an impo an componen o he pa allel p og am de elopmen and uning cyqle. To ob ain he aw pe o mance da a o an applica i~n, an ins omen ed e sion o he applica ion is un wi h p obes ha ake measu es o speci ic e en s o pe o~ance indica o s (i.e. ha dwa e coun e s, sub ou ines, pa allel loops). The ob ained ace da a can be summa ized on-line by he acing ool. Mo e o en, howe e , i is s o ed ili1 ace iles o o -line analysis. We ocus ou in e es in 1I acing packages o pa allel p og ams, whe e all he ac~ui ed da a is s o ed in ace iles o a de ailed analysis a !a la e ime. T acing pa allel p og ams wi h such aci~ ools easily leads o huge ace iles wi h hund ~s o Megaby es, which has se e al p oblems conc!e ning s o age, isualiza ion and analysis o such aces. We p opose a ace-scaling agen o acing Qols o pa allel applica ions. In un ime he agen lea$ he pe iodic s uc u e in he applica ion beha io exhibi~ed by many scien i ic p og ams. The cap ion o he appljca ion beha io allows s o ing only he non-i e a i e ace in o ma ion, which d as ically educes he s o age I This wo k has been suppo ed by he Spanish Minis I)' o Science and Technology unde TIC2001 -0995-CO2-01 and by he Eu opean Union (FEDER). P oceedings o he 14 h IEEE In e na ional Con e ence on Tools wi h A i icial In elligence (ICTAI’02) 1082-3409/02 $17.00 © 2002 IEEE Au ho ized licensed use limi ed o: IEEE Xplo e Cus ome . Downloaded on Oc obe 13, 2008 a 08:54 om IEEE Xplo e. Res ic ions apply. 2.2. Rela ed wo k he ace ile. The analysis o such a educed ace allows uning he main i e a i e body o he applica ion. 3. T ace-scaling agen 3.1 Recogni ion o i e a i e pa e ns The mos equen app oach o es ic he size o he ace in cu en p ac ice is o inse calls in o he sou ce code o he applica ion o s a and s op he acing. Sys ems such as Vampi T ace [6], VGV [4 , and OMPI ace [I] p o ide his mechanism. This a oach equi es he modi ica ion o he sou ce code, whi h may no always be a ailable o he pe o mance analys .E en i he sou ce code is a ailable, i is necessa y o e a ce ain unde s anding o i be o e being able o p ope ly inse he acing con ol calls. The Pa adyn p ojec [5] de eloped an ins ume a ion echnology (Dynins ) h ough which i is poss le o dynamically inse and ake ou p obes in a ing p og am. Al hough no e o is made o au om ically de ec pe iods, he me hodology behind his app oa h also elies on he i e a i e beha io o applica ions. The au oma ic pe iodici y de ec ion idea we p esen his pape could be use ul inside such a dynamic analy is ool o p esen o he use he ac ual s uc u e he applica ion. In IBM UTE [7], an in e media e app oach is o lowed o pa ially ackle he p oblem, which la ge aces ose o he analysis ool. The acing acili y can gene a huge aces o e en s, con aining in o ma ion wi h a lo o de ail down o he le el o con ex swi ches and global ys em ac i i ies. Then, il e s a e used o ex ac a a e ha ocuses on a speci ic applica ion, summ izing in o ma ion in eco d o ma s mo e amena le o isualiza ion and be e desc ibing he appl ca ion beha io . To p ope ly handle he as access o s eci ic egions o a la ge ace ile he SLOG o ma (s alable log ile o ma ) has been adop ed. Using a ame in x he Jmnpsho isualiza ion ool [8] imp o es he acce s ime o ace da a. The ace o he applica ion is a da a s eam con aining he alues o se e al pa ame e s. I he applica ion con ains loops, hen i has segmen s wi h pe iodic pa e ns. We apply a pe iodici y de ec ion algo i hm o he da a s eam in o de o segmen he da a s eam in o pe iodic pa e ns. The used algo i hm is ame based and equi es a ini e leng h o pas da a alues o compu e he pe iodici y. We implemen he pe iodici y de ec o om [3] in he ace-scaling agen in o de o pe o m he au oma ic de ec ion o i e a i e s uc u es in he ace. The s e3ln o pa allel unc ion iden i ie s om he ace is he inpu . The ou pu o he agen is he indica ion whe he pe iodici y exis s in he da a s eam and i s pe iod leng h. The algo i hm used by he pe iodici y de ec o is based on he dis ance me ic gi en by he equa ion ,V-I d(m) = sign Ii x(i)- x(i -m)1 1=0 ( 1 ). In equa ion (I) N is he size o he da a window, m is he delay (O<m<M), M<=N, x[i] is he cu en alue o he da a s eam, and d(m) is he alue compu ed o de ec he pe iodici y. I can be seen ha equa ion (1) compa es he da a sequence wi h he da a sequence shi ed m samples. Equa ion (1) compu es he dis ance be ween wo ec o s o size N by sumJning he magni udes o he L I -me ic dis ance o N ec o elemen s. The sign unc ion is used o se he alues d(m) o 1 i he dis ance is no ze o. The alue d(m) becomes ze o i he da a window con ains an iden ical pe iodic pa e n wi h pe iodici y m. I he pe iodici y m in he da a s eam is se e al magni udes less han he size N o he da a window, hen he alue d(m) may become ze o o mul iples o m. On he o he hand i he pe iodici y m in he da a s eam is la ge han he da a window size N, hen he de ec o canno cap u e he pe iodici y. The pe iodici y leng h we ound in he used applica ions was usually small (be ween 5 -20) and less han 300. Fo an unknown da a s eam, he window size N o he pe iodici y de ec o can be se ini ially o a la ge alue, in o de o be able o cap u e po en ially la ge pe iodici ies. Once a sa is ying pe iodici y is de ec ed, he window size can be educed dynamically. 2.3 Ou app oach ~Ou app oach o he scalabili y p oblem o aci is o adap dynamically he aced ime. We p opose a ace- scaling agen , which lea ns in un ime he s uc u e o he applica ion. I au oma ically de e mines he le an acing in e als, which a e su icien o cap e he applica ion beha io . Wi h he ace-scaling age i is possible o ace only one o se e al i e a ions he dynamically de ec ed epe i i e pa e n in he appl ca ion beha io . Ou app oach does no equi e limi " 9 he g anula i y o acing, no he numbe o pa ame e s ead a e e y acing poin , no he p oblem size. Due o he dynamic in e cep ion o he calls o un ime lib a ie in he acing ool, ou implemen a ion does no equi e he sou ce code o he applica ion o achie e he scal ace. In un ime he edundan ace in o ma ion is ide i ied and only he non-i e a i e applica ion beha io is s ed in P oceedings o he 14 h IEEE In e na ional Con e ence on Tools wi h A i icial In elligence (ICTAI’02) 1082-3409/02 $17.00 © 2002 IEEE Au ho ized licensed use limi ed o: IEEE Xplo e Cus ome . Downloaded on Oc obe 13, 2008 a 08:54 om IEEE Xplo e. Res ic ions apply. 3.2. Implemen a ion The new sample o e w i es he column con aining he oldes alues wi h he new dis ance. The implemen a ion wi h ci cula lis s a oids mo ing he da a alues. The numbe o ope a ions a ins an i a e educed, which leads o a small o e head o his implemen a ion. 3.3. OpenMP and acing ool in eg a ion The s uc u e o OpenMP based pa allel applica ions usually i e a es o e se e al pa allel egions. Fo each pa allel di ec i e he mas e h ead in okes a un ime lib a y passing as a gumen he add ess o he ou lined ou ine. The acing mechanism in e cep s he call and ob ains a s eam o pa allel unc ion iden i ie s. This s eam con ains all execu ed pa allel unc ions o he applica ion, bo h in pe iodic and non-pe iodic pa allel egions. We ha e implemen ed he ace-scaling agen in he OMPI ace acing ool [I]. The acing ool gene a es ace iles, which consis o e en s (ha dwa e coun e alues, pa allel egions en y/exi , use unc ions en y/exi ) and h ead s a es (compu ing, idle, o k/join). ull ace da a s eam i e a i e pa e ns scaled ace iLo ali.o bcha.;o DO l aoc i. w iLIOD 11111111 IIIIIIIIIIIIIIIIII 111111111 ~ Figu e I. In e ac ion o he agen in he acing ool. In Figu e 1 he in e ac ion be ween he ins umen ed applica ion, he acing ool, and he agen is illus a ed. I can be seen ha he agen ecei es a da a s eam om he acing ool. The da a s e~ con ains he alues o a aced pa ame e such as he iden i ie s o he execu ed unc ions in pa allel egions. The agen lea ns he applica ion beha io . Ha ing his indica ion he acing ool knows, which is he non-i e a i e in o ma ion o w i e o he ace ile. In ou implemen a ion o equa ion ( I ), we s o e ~ ini e numbe o p e ious da a alues o he da a Is eam including he mos ecen alue in a da a ec o . ~ n his da a ec o he algo i hm pe o ms pe iodici y de c ion. This da a ec o can be implemen ed as a FIFO b e o leng h M+N. This ype o implemen a ion uses ~ leas amoun o memo y, bu equi es a highe numbe o ope a ions a e e y ins an i han o he implemen a ions. Applying equa ion (I) on he da a ec o equi ,s M x N ope a ions o compu e he alues o d(m) a he ins an i o he da a s eam. I can be obse ed, howe e , ha some ope a ions a e done wi h he same da a alues ~e e al imes a di e en ins an s i. The p e iously co pu ed dis ances be ween ec o elemen s could be s o d o educe he numbe o compu a ions made by he algo i hm a ins an i. The e is a ade-o be ween he nwnbe o compu a ions made a ins an i. and he amoun o memo y needed by he algo i hm. In o de o educe he amiun o compu a ion we implemen a FIFO o ganized ma ix o size MxN whe e p e iously compu ed dis anc s a e s o ed. Using his dis ance ma ix we compu e ~ each ins an i he alue o di(m) o all alues o m, whd; e x(i) is he alue o he da a alue a he cu en samplel i, and x(i-m) is he da a alue ob ained m samples be o b. The compu ed alues o d;(m) a e w i en in he 9olumn co esponding o he ins an i in he ma ix. i In case o using he dis ance ma ix o s o e p eJiously compu ed dis ances, hen only M ins ead o j x N ope a ions need o be made a ins an i o ob ain e new dis ances di(m). The alue o d(m) o all alue m is ob ained by summing he elemen s o each aw 1o he dis ance ma ix, i.e. he p e iously compu ed di ances and each mos ecen dis ance d;(m). This means ha a e e y ins an i new alues a e w i en in a columnlo he dis ance ma ix, and d(m) is compu ed as he sum lo he alues in each aw using he p e iously compu ed ~alues o he o he columns. Using he dis ance ma ix he l ize o he da a ec o can be educed o M, since a ins an i only he dis ances be ween he ne~l da a alue and he M I pas da a alues need o be compu ed. I can be s en in equa ion ( I) ha i he da a window size N and h delay M is inc eased, la ge i e a i e s uc u es can be de ec ed. Then, he numbe o compu a ions o ob ain d( ) also inc eases. Howe e , when inc easing N and M a d he p e iously compu ed dis ances a e e-used, h n he inc ease o ope a ions is only linea . i In o de o educe he numbe o shi s o hel FIFO ope a ions, he da a s uc u es o he pe iodici y de ec o concep ually wo king as FIFO o ganized ma ix and FIFO o ganized da a ec o a e p og ammed as ci cula lil s. A each ins an i he poin e o he cu en lis elemen shi s by one such ha i poin s o he oldes alues in e lis . P oceedings o he 14 h IEEE In e na ional Con e ence on Tools wi h A i icial In elligence (ICTAI’02) 1082-3409/02 $17.00 © 2002 IEEE Au ho ized licensed use limi ed o: IEEE Xplo e Cus ome . Downloaded on Oc obe 13, 2008 a 08:54 om IEEE Xplo e. Res ic ions apply. 4. Applica ions o scaled acing 4.3. Imp o emen o he ease o isualiza ion 4.1. Expe imen al se up We ace he applica ions gi en in Table 1 i Fou applica ions om he NAS benchma k sui e: E (cl~ss A), I Lu (class A), Cg (class A) and Sp (class A); aI1d i e applica ions o he SPEC95 sui e: Swim, Hyd o2dj Apsi, Tomca , and Tu b3d, all wi h e da a se . ! All expe imen s a e ca ied ou on a Silicon G~aphics O igin 2000. The OpenMP applica ions a e execu ~d in a dedica ed en i onmen wi h 8 CPUs. We con igq e he ace-scaling agen such ha a e ha ing de ec,ed 10 i e a i e pa allel egions i s ops w i ing ace da al o he ile un il i obse es a new p og am beha io~. The pa ame e s con ained in he ace ile a e he h eaq s a es and OpenMP e en s, which include wo ha dwa e coun e s. Conside ing he ull ace in Figu e 4 (see mal page) a i s isual pe cep ion o he p og am beha io can be qui e misleading. Fo example, i seems ha he e is a lo o o ~join ac i i y in he i s h ead (whi e colo ) while his is only an e ec o he display p ecision. The eason is ha a he scale ha had o be used o display he whole ace, each pixel ep esen s a la ge ime in e al (152 ms) wi hin which one h ead can pe o m many changes o ac i i y. In Figu e 5 (see inal page) we can easily iden i y ha he e is a pe iodic pa e n (pe iod bounda ies agged wi h lags). I can be obse ed ha a e a ce ain numbe o epe i ions his pa e n changes and ha a new pe iodic pa e n is hen epea ed. The di ec look a he ull ace o Figu e 4 ha dly e eals ha he e is a special beha io in he middle pa . The lags in Figu e 5 iden i y he pe iod. Wi h he scaled ace i is immedia e o zoom o an adequa e le el o see he ac ual pa e n o beha io . In he isualiza ion o he scaled ace, he i e a i e ace in o ma ion is no shown (Figu e 5 black a ea), since he acing mechanism did no w i e i o he ace ile. Table I. E alua ed benchma ks. Benchma ks --- Applica ion NAS E NAS Cg NAS Lu NAS Sp Apsi Hyd o2D Swim Tomca Tu b3d NAS benchma ks 4.4. Reduc ion o he ace ile size We examine how much he mce ile size educes when using he ace-scaling agen . Figu e 2 shows he size o he ace iles o he NAS and SPEC95 benchma ks ob ained wi h and wi hou using he agen . I can be seen ha wi h scalable acing he ace iles a e educed signi ican ly. The NAS Lu ace ile, o ins ance, educes om I73 Mb o 8 Mb, which is a educ ion o 95%. Had we aced less han IO i e a ions, he ace size would educe mo e. SPEC95 p benchma ks 4.2. Applica ion s uc u e iden i ica ion - .0 ~ - . . -0) c: ~ Q) u ~ ... The ace-scaling agen allows inse ing in o$a ion abou he de ec ed applica ion s uc u e in hei ace eco ds, which indica es he s a /end o an i ~ a i e pa e n. This in onna ion is highly use ul o he I/nalys because one o he i s ac i i ies when acing a la g ace is o zoom down, ying o iden i y an a ea o !a ew pe iods ha can be aken as e e ence o loo~ng a de ails. The acing ool w i es hese e en s indica ing pe iodic pa e ns o he ace e en i he w i ing ! o all o he ace in o ma ion is suspended. In Figu e 3 (see mal page) wo i e a i e egionsi o he NAS E benchma k wi h hei h ead s a es a e shown. The bounda ies o he i e a i e egions a e ep esen ed as lags, which e eal he applica ion s uc u e. The numbe o pe iodic pa e ns and hei du a ion can easJly be comDu ed om he De iodici e en in he ace ilei P oceedings o he 14 h IEEE In e na ional Con e ence on Tools wi h A i icial In elligence (ICTAI’02) 1082-3409/02 $17.00 © 2002 IEEE Au ho ized licensed use limi ed o: IEEE Xplo e Cus ome . Downloaded on Oc obe 13, 2008 a 08:54 om IEEE Xplo e. Res ic ions apply. e icien analysis. We ha e p oposed a ace-scaling agen , which allows s o ing da a o a comple e analysis while achie ing a small ace ile. We ha e implemen ed he agen , which lea ns he applica ion beha io in un ime and allows s o ing only he non-i e a i e ace da a. We ha e shown ha he size o such a scaled ace ile becomes signi ican ly educed, while in he aced in e al he ele an applica ion beha io is cap u ed. We obse ed ha he scaled aces a e easy o handle by isualiza ion ools and he scaled ace le s he analys us e obse e ele an applica ion beha io such as he applica ion s uc u e. Ou implemen a ion o he ace- scaling agen has a small o e head and i is used in un ime. The scaled ace can subs i u e he ull ace in se e al pe o mance analysis asks, since i allows he pe o mance analys o each he same conclusions on he applica ion pe o mance as when using he ull ace. - ~.Full ace .10l e a io s -- .c ~ .c -C) c ~ Qj ~ .. ... NAS lu Hyd o2D Figu e 2. Compa ison o he ace ile size wi~h ull and scalable acing. 5. Discussion 7. Re e ences The o e head o an implemen a ion is an impo an pe o mance ac o o eal- ime ools. In [2] w~ ha e e alua ed he o e head p oduced by he ace-$caling agen . I was obse ed ha he o e head in odu4ed by acing is small in e ms o he execu ion im The o iginal acing ool adds 1% -3% o he execu io ime. Wi h he ace-scaling agen , he o e head is 3% -6 0. In applica ions wi h a pe iodic pa e n ",'e ex ec o each he same concluysions on pe o mance I when analysing a subse o he i e a ions, i.e. he scaled hce. In [2] we ha e compa ed he pe o mance indices coTpu ed om he scaled and ull aces. Ou esul s show ~ he same pe o mance conclusions can be ob ained I when analysing he scaled ace o he applica ions. i The agen lea ns he applica ion beha io he s eam o unc ion iden i ie s. I could be possible he agen de ec s i e a i e beha io in he execu ed un ions, bu a he same ime he pe o mance o he o he i dices (cache misses, TLB misses, ...) could di e signi can ly om one i e a ion o ano he . I such a case occu ~ in an isola ed pa allel egion, he agen would no de e~ his si ua ion. I Ou ool elies on he i e a i e beha io O apPli ions, whe e loops a e execu ed many imes. Many sc. n i ic applica ions ha e such a s uc u e. The s udied N S and SPEC95 benchma ks, which mos ly pe o m n e ic compu a ions, exhibi i e a i e applica ion beha ~o . In case o ha ing he ace-scaling agen ac i a e4 wi h ano he class o applica ions, which a e non-i e a i e, simply no pe iodic beha io would be de ec ed and he whole ace would be w i en o he ile. [1] J. Caube , J. Gimenez, J. Laba a, L. DeRose, J. Ve e . "A Dynamic T acing Mechanism o Pe o mance Analysis o OpenMP Applica ions." In In e na iona/ Wo kshop on Open, 1P App/ica ions and Too/s (WO, 1PAT 2001), July 2001, pp. 53-67. [2] J. Caube , F. F ei ag, J. Laba a. "Compa ison o scaled and ull aces o OpenMP applica ions." Tech. Repo UPC-DAC-200 1-31. [3] F. F ei ag, J. Co balan, J. Laba a. "A dynamic pe iodici y de ec o : Applica ion o speedup compu a ion." In In e ma iona/ Pa a//e/ and Dis ibu ed P ocessing Symposium (IPDPS2001), Ap i12001. [4] J. Hoe linge , B, Kuhn, W. Nagel, P. Pe e sen, H. Rajic, S. Shah, J. Ve e , M. Voss, and R. Woo. An In eg a ed Pe o mance Visualize o MPl/OpenMP P og ams. In In e na iona/ Wo kshop on Open, 1P Applica iom and Tools (WO, 1PAT2001), July 2001, pp. 40-52. [5] B. P, Mille , M. D. Callaghan. The Pa adyn Pa allel Pe o mance Measu emen Tools. IEEE Compu e 28(11): 37-46, No embe 1995. [6] Pallas: Vampi ace. Ins al/a ion and Use 's Guide, h m:llwww,12allas.de [7] C. E. Wu, A, Bolma cich, M. Sni , D. Woo on, F. Pa pia, A. Chen, E. Lusk and w, G opp. F om T ace Gene a ion o Visualiza ion: A Pe o mance F amewo k o Dis ibu ed Pa allel Sys ems. In P oceedings o Supe Compu ing (SC 2000), No embe 2000. [8] 0. Zaki, Eo Lusk, W, G opp, and D. Swide , Towa d scalable pe o mance isualiza ion wi h Jwnpsho . In In e na iona/ Jou na/ o High Pe o mance Compu ing App/ica iom. 13(2): pages 277-288, 1999. 6. Conclusions We ha e desc ibed he some scalabili y p obl~ms o acing in cun-en pe o mance analysis ools an~ why hese a e a p oblem o s o age, isualiza io~, and P oceedings o he 14 h IEEE In e na ional Con e ence on Tools wi h A i icial In elligence (ICTAI’02) 1082-3409/02 $17.00 © 2002 IEEE Au ho ized licensed use limi ed o: IEEE Xplo e Cus ome . Downloaded on Oc obe 13, 2008 a 08:54 om IEEE Xplo e. Res ic ions apply. Figu e 3. Visualiza ion o he h ead s a es in he NAS B applica ion in 2 i e a i e pa allel egions. Ligh colo =idle, da k colo =compu ing. Figu e 4. Visualiza ion o he whole Hyd o2D lexecu ion ace ( ull ace). Figu e 5. Visualiza ion o he Hyd o2D execu ion wi h scaled acing (scaled ace). P oceedings o he 14 h IEEE In e na ional Con e ence on Tools wi h A i icial In elligence (ICTAI’02) 1082-3409/02 $17.00 © 2002 IEEE Au ho ized licensed use limi ed o: IEEE Xplo e Cus ome . Downloaded on Oc obe 13, 2008 a 08:54 om IEEE Xplo e. Res ic ions apply.