A motion and edge adaptive interlaced-to-progressive conversion using fuzzy logic-based systems

A mo ion and edge adap i e in e laced- o-p og essi e con e sion using uzzy logic-based sys ems P. B ox Ins i u o de Mic oelec ´onica de Se illa (CSIC) and Uni e si y o Se ille (Spain) b o[email p o ec ed] I. Ba u one Ins i u o de Mic oelec ´onica de Se illa (CSIC) and Uni e si y o Se ille (Spain) [email p o ec ed] S. S´anchez-Solano Ins i u o de Mic oelec ´onica de Se illa (CSIC) Se ille (Spain) san[email p o ec ed] Abs ac This pape p esen s an algo i hm o ideo de-in e lacing. The app oach uses h ee uzzy logic-based sys ems o adap he in e pola ion s a egy o he p esence o mo ion and edges. Fu he mo e, he algo i hm is able o deal wi h any kind o TV ma e ial independen ly o he sou ce used o acqui e he scene. Ex ensi e simula- ions o s anda d and eal sequences p o e he e iciency o he p oposed algo i hm. Keywo ds: Video de-in e lacing, uzzy logic-based sys em, mo ion adap i e, edge adap i e. 1 In oduc ion In e lacing was in oduced by he TV commu- ni y since i p o ides an e ec i e educ ion o he ideo bandwid h. I educes he band- wid h a hal since only he e en o odd lines ha compose a ame a e al e na i ely ans- mi ed. In e lacing is cu en ly used by all he analog TV s anda ds (PAL, NTSC and SECAM) and also, by some o he mo e mod- e n digi al ansmissions [1]. Recen ly, he e is an inc easing need o a p o- g essi e scanning o ma a he ecei e side o TV signal. Many de ices such as mode n dis- plays (LCDs, Plasma), DVDs, and p ojec o s, wo k wi h p og essi e ma e ial and inco po- a e an embedded chip ha implemen a de- in e lacing algo i hm. I consis s o con e - ing in e laced ideo in o a p og essi e o m by in e pola ing he non- ansmi ed lines. Se - e al ea u es o he pic u e like he p esence o mo ion and edges could complica e his ask. Many de-in e lacing algo i hms ha e been p oposed in he li e a u e du ing he las yea s [2]. Basically, hey can be classi ied in o wo ca ego ies: mo ion (MC) and non- mo ion compensa ed (non-MC) algo i hms. MC echniques look o a mo ion ec o in each pixel o block o pixels o he image and achie e he bes esul s in mo ing a eas. How- e e , he compu a ional cos in ol ed in he calcula ion o he mos app op ia e mo ion ec o is qui e high. An al e na i e among non-MC algo i hms a e he mo ion adap i e de-in e lacing echniques [3]-[6]. As i s name indica es, his kind o al- go i hms es ima es he le el o mo ion in he image and adap he in e pola ion s a egy acco ding o i . I he e is no mo ion o he le el o mo ion is ba ely app eciable hen he empo al neighbo s a e sui able o pe o m he in e pola ion. On he con a y, when he le el o mo ion is high a spa ial in e pola o is chosen o in e pola e he new pixel. The e iciency o mo ion adap i e algo i hms elies on he quali y o he mo ion de ec o . P imi i e app oaches use he di e ence be- ween pixels wi h he same spa ial coo di- na es om wo consecu i e ames o measu e mo ion, and a c isp ansi ion be ween he empo al and spa ial in e pola o [3]. How- e e , hey a e a o achie e good esul s in icky pa s o he image, which con ain high con as de ailed a eas, high le el o mo ion, L. Magdalena, M. Ojeda-Aciego, J.L. Ve degay (eds): P oceedings o IPMU’08, pp. 1175–1182 To emolinos (M´alaga), June 22–27, 2008 noise and/o a high numbe o edges. To imp o e he obus ness o he mo ion de- ec o se e al p oposals ha e been p esen ed in he li e a u e du ing he las yea s [2], [4]. Some au ho s combine he ou pu o se e al mo ion de ec o s [2], whe eas o he s apply il- e ing echniques o ield di e ence signal [4]. O he au ho s imp o e he pe o mance o mo ion adap i e de-in e lacing algo i hms [4]- [6]. In [4], he c isp ansi ion be ween he empo al and spa ial in e pola o is subs i- u ed by a so ansi ion. In his sense, se - e al op ions a e p o en in [4] such as a linea o a s ep piecewise ansi ion. O he al e na- i e is o use uzzy logic and o apply di e - en heu is ic ules wi h app oxima e le els o unce ain y, which implici ly pe o m a non- linea il e ing [5]-[6]. This pape desc ibes a new mo ion adap i e de-in e lacing as esul o a wo k de eloped du ing he las yea s. The combina ion o he spa ial and empo al in e pola o is ca ied ou by a uzzy sys em (F S1), whose inpu is a bi-dimensional con olu ion o ield di e ence signal. Fu he mo e, he spa ial and empo al in e pola o s a e also calcula ed by wo uzzy sys ems: a second uzzy sys em (F S2) p o- ides he empo al in e pola o , which is ca- pable o dealing wi h any kind o ideo ma- e ial, and a hi d uzzy sys em (F S3) is able o adap he spa ial in e pola ion o edges. a1 a2 a b c c1 c2 FS3 mo ion IS dissimila i y FS2 ou pu FS1 IT Figu e 1: Block diag am o he p oposed al- go i hm. 2 Desc ip ion o he algo i hm Figu e 1 shows a desc ip i e diag am o he implemen ed algo i hm. The ollowing sub- sec ions desc ibe he h ee uzzy sys ems used in he p oposed algo i hm. 2.1 Fuzzy sys em o combine he in e pola o s wi h he p esence o mo ion (F S1) Since cu en TV s anda ds wo k wi h ideo coding algo i hms whe e luminance compo- nen con ains mo e in o ma ion han ch omi- nance componen s [1], ou s udy is only de el- oped o his unique componen . Howe e , i s ex ension o colo images is simple and di ec by applying he inal in e pola ion exp ession o he colo componen s. Ou p oposal uses as inpu alue he bi- dimensional con olu ion o ield di e ence sig- nals ha can be ma hema ically exp essed as ollows: mo ion =ΣCj,iHi,j ΣCj,i = =(1 2 1) (H1,1H1,2H1,3)T 4(1) whe e Cj,i a e he alues o he weigh s and Hi,j a e desc ibed by he ollowing di e ences o luminance alues (see Figu e 2(a)): H1,1=|B0−B|(2) H1,2=|X0−X|(3) H1,3=|E0−E|(4) Di e en weigh s and sizes o ma ix Hha e been s udied o achie e a good ade-o be- ween he compu ing esou ces and he qual- i y o mo ion measu emen [7]-[8]. As can be seen in exp ession (1), he selec ed con olu- ion only includes neighbo s in e ical di ec- ion since a wide numbe o ideo sequence simula ions shown a non-decisi e in luence o ho izon al neighbo s o measu e he le el o mo ion. Unlike he p oposals in [5]-[6], which use ou ields, ou algo i hm educes he empo al ape u e up o h ee ields as shown Figu e 1176 P oceedings o IPMU’08 B0 X0 E0 X B E Xn ( -1) In e pola ed line T ansmi ed line ( ) Sequence o de ( +1) (a) 3 d mo ion is M X=λIT+δIS 2nd mo ion is L X=IS 1s mo ion is S X=IT I an eceden consequen 1 0 S (small)M (medium)L (la ge) µmo ion mo ion (b) (c) Cu en pixel Figu e 2: (a) Pixels in ol ed in he calcula ion o he bi-dimensional con olu ion. (b) Rulebase o he F S1. (c) Membe ship unc ions used in F S1. 2(a). The in e pola ed alues calcula ed in he p e ious ield (B0, E0) a e necessa y o e alua e he mo ion alue in exp ession (1). The spa ial in e pola o (IS) is employed o calcula e he i s p og essi e ame. The in luence o mo ion in selec ing he kind o in e pola ion is e alua ed by conside ing h ee ules ha a e linguis ically exp essed as ollows: 1. I mo ion in he cu en pixel is small (S), he mos adequa e in e pola ed alue is ob ained by applying a empo al in e - pola ion (IT). 2. I mo ion in he cu en pixel is la ge (L), he bes esul is ob ained by pe o ming a spa ial in e pola ion (IS). 3. I mo ion in he cu en pixel is medium (M), hen he alue is be e calcula ed by applying a linea combina ion o he empo al and spa ial in e pola o s (λIT+ δIS). This ulebase is summa ized in he Table o Figu e 2(b). The uzzy concep s small, la ge and medium used in he ules a e modeled ac- co ding o he membe ship unc ions shown in Figu e 2(c). Using he Fuzzy Mean as de uzzi- ica ion me hod he new pixel alue is calcu- la ed as ollows: X=α1IT+α2IS+α3(λIT+δIS) (5) whe e αiis he co esponding ac i a ion de- g ee o each ule in he Table o Figu e 2(b). Th ee deg ees o mo ion (small, medium and la ge) a e conside ed in his uzzy sys em. A - e analyzing up o i e deg ees o mo ion [9], he ulebase wi h h ee ules has been selec ed since i p o ides he mos a ac i e solu ion in e ms o ha dwa e esou ces and quali y o he in e pola ed image. 2.2 Fuzzy logic-based sys em o he empo al in e pola ion (F S2) In o de o unde s and he s a egy imple- men ed in F S2 o ob ain he empo al in e - pola o , i is necessa y o e iew he o igin o ma e ial. I he sequence was eco ded by a ideo came a a a pic u e a e o 50 Hz (PAL) o 60 Hz (NTSC), he h ee ields o he ape u e a e di e en in mo ing a eas o he image (di e en numbe s in Figu e 3(a)). Howe e , i he ma e ial was egis e ed wi h a cine-came a he pic u e a e is 24 Hz and a con e sion o ilm ma e ial is necessa y o dis- play i on TV. The con e sion o adap bo h pic u e a es basically consis s o epea ing he ields wice ( o achie e 50 Hz), o wice and h ee imes al e na i ely ( o achie e 60 Hz) as i is shown in Figu e 3(b). This p o- cess is known as pull-down 2:2 and pull-down 3:2, espec i ely. Since he empo al ape u e o his app oach would be composed om ilm ma e ial ( o in- s ance 2-e en, 2-odd, 3-odd), wo o he ields in he ape u e has o come om he same o iginal ame. The de ec ion o hese cases is e y in e es ing due o wo easons. Fi s ly, he isen p esence o ilm and hyb id ma e ial P oceedings o IPMU’08 1177 123 Sou ce: Telecine 24 Hz 112233 50 Hz 50 Hz 123456 Sou ce: Video came a (a) (b) In e laced ma e ial e en odd e en odd e en odd In e laced ma e ial e en odd e en odd e en odd Figu e 3: (a) Video sequence. (b) Sequence o ilm ma e ial. 1 0 S (small)L (la ge) μdissimila i y dissimila i y (a) (b) B0 E0 X B E ( -1) ( ) Sequence o de X0 2nd dissimila i y is L IT = Xn 1s dissimila i y is S IT = X0 I an eceden hen consequen Figu e 4: (a) Rulebase o he F S2. (b) Mem- be ship unc ions used in F S2. on TV and secondly, a pe ec de-in e lacing can be achie ed by copying his in o ma ion om he epea ed ield in he ape u e a a ex- pense o a minimal cos (i he epea ed ield is co ec ly de ec ed in he ape u e). A simple uzzy sys em is p oposed ha is able o deal wi h ilm ma e ial. I selec s he mos adequa e empo al in e pola ion de- pending on dissimila i y signal be ween wo consecu i e ields, gi en by he ollowing ex- p ession: dissimila i y =|B−B0|+|E−E0| 2(6) The heu is ic knowledge o his uzzy sys em is exp essed by means o he ollowing linguis- ic ules: 1. I dissimila i y be ween he ields ( -1) and ( ) is small (S), he mos adequa e in e pola ed alue is ob ained by selec - ing he pixel alue in he p e ious ield a he same spa ial posi ion (X0)(see Fig- u e 4(a)). 2. On he con a y, i dissimila i y is la ge (L), he pixel alue in he p e ious ield is no a good choice and is be e o be on he pixel in he nex ield (Xn)(see Figu e 4(a)). Table in Figu e 4(a) summa izes he ulebase o his second uzzy sys em. The shape o membe ship unc ions o model he uzzy con- cep s small and la ge a e shown in Figu e 4(b). The ou pu o his uzzy sys em is gi en by he ollowing exp ession: IT=β1X0+β2Xn(7) whe e βiis he ac i a ion o each ule in he Table o Figu e 4(a). 2.3 Fuzzy logic-based sys em o he spa ial in e pola ion (F S3) F S3pe o ms a sma in e pola ion among pixels in he spa ial neighbo hood. The heu is ic knowledge de eloped in he uzzy ulebase adap s he in e pola ion s a egy ac- co ding o he p esence o edges in he pic- u e. To de ec edges he ollowing di e ences 1178 P oceedings o IPMU’08 XC1 BC A1 F1 A D1DEF ( ) Sequence o de (a) 5 h a1is VL and a is VL and b is L IS=(C1+D1)/2 and c is Land c1is S 3 d a is VS and b is L and c is VS IS=(A+F+C+D)/4 2nd a is L and b is L and c is S IS=(C+D)/2 6 h o he wise IS=(B+E)/2 4 h a1is S and a is L and b is L IS=(A1+F1)/2 and c is VL and c1is VL 1s a is S and b is L and c is L IS=(A+F)/2 I an eceden hen consequen 1 0 VS ( e y small) VL ( e y la ge) μa a (b) (c) a=|A-F| a1=|A1-F1| b=|B-E| c=|C-D| c1=|C1-D1| SL Figu e 5: (a) Pixels in ol ed in he spa ial in e pola o . (b) Rulebase o he F S3. (c) Membe - ship unc ions used in F S3. abc a1c1 26.5º 45º 135º 153.43º Figu e 6: Di ec ions e alua ed by he F S3. o pixel alues along i e di ec ions a e calcu- la ed (see Figu e 5(a) and Figu e 6): a1=|A1−F1|(8) a=|A−F|(9) b=|B−E|(10) c=|C−D|(11) c1=|C1−D1|(12) The ollowing knowledge is employed o es i- ma e he edge adap i e in e pola ion: 1. I he e is a small (S) di e ence in di- ec ion a, and i band ca e la ge (L), hen an edge could be in di ec ion aand he bes solu ion is o apply he a e age be ween he wo pixels ha de ines adi- ec ion. 2. I he e is a small (S) di e ence in di- ec ion c, and i band aa e la ge (L), hen an edge could be in di ec ion cand he bes solu ion is o apply he a e age be ween he wo pixels ha de ines cdi- ec ion. 3. I he e is a e y small (VS) di e ence in di ec ions aand c, and a la ge (L) di - e ence in di ec ion b, nei he he e is an edge no e ical linea in e pola ion pe - o ms well; he bes op ion is a linea in- e pola ion be ween he neighbo s wi h small di e ences: A, C, D, F. 4. An edge is clea in di ec ion a1no only i a1is small (S), bu also i aand ba e la ge (L) and cand c1a e e y la ge (VL). Then he spa ial in e pola ion is calcu- la ed by applying he a e age be ween he wo pixels ha de ines a1di ec ion. 5. An edge is clea in di ec ion c1no only i c1is small (S), bu also i band ca e la ge (L) and aand a1a e e y la ge (VL). Then he spa ial in e pola ion is calcu- la ed by applying he a e age be ween he wo pixels ha de ines c1di ec ion. 6. O he wise, a e ical linea in e pola ion would be he mos adequa e. Table o he Figu e 5(b) summa izes he ule- P oceedings o IPMU’08 1179 Table 1: A e age PSNR alues (in dBs) using ideo sequences. SEQUENCE Missa Pa is T e o Salesman News Mo he Ca phone FORMAT CIF CIF CIF CIF QCIF QCIF QCIF Line Doubling 36.44 23.61 31.05 29.75 25.18 31.81 28.25 Line A e age 40.47 26.67 35.04 33.53 29.25 35.94 32.61 ELA 3+3 39.49 25.53 34.11 32.11 26.63 35.39 32.65 ELA 5+5 38.56 24.64 33.31 30.17 25.92 34.2 31.51 Field Inse ion 38.36 29.86 34.36 36.17 33.13 36.14 30.34 VT 2 ields 40.25 30.73 36.61 36.54 35.46 39.61 34.08 VT 3 ields 40.52 31.37 37.16 36.95 35.67 40.89 34.54 Technique in [5] 40.01 33.12 35.38 37.62 34.73 39.49 32.27 Technique in [6] 40.18 35.28 36.69 38.29 37.51 41.87 34.78 P oposal 40.81 35.87 37.63 38.35 38.78 42.11 35.09 base o his second uzzy sys em. F om he analysis o hese ules, we can see ha a highe numbe o an eceden s a e used in he ules ha e alua e a1and c1di ec ions, since a ein o cemen is necessa y o a oid he de- ec ion o alse edges when he sys em wo ks wi h 5+5 pixels in he neighbo hood. The shape o membe ship unc ions o model he uzzy concep s small, la ge, e y small and e y la ge a e shown in Figu e 5(c). The ou - pu o his uzzy sys em is ob ained by apply- ing he Fuzzy Mean as ollows: IS=χ1(A+F 2) + χ2(C+D 2)+ +χ3(A+C+D+F 4) + χ4(A1+F1 2)+ +χ5(C1+D1 2) + χ6(B+E 2) (13) whe e χiis he ac i a ion o each ule in he Table o Figu e 5(b). 3 Simula ion esul s The pe o mance o he p oposed algo i hm has been analyzed by de-in e lacing se e al ideo sequences. They can be di ided in o wo ca ego ies: a i s g oup o s anda d ideo sequences and a second one o eal ilm se- quences. The ideo sequences conside ed ha e widely been used as benchma ks in ideo p ocessing applica ions. A e ob aining he in e laced ideo da a om hese p og essi e sequences by elimina ing lines, se e al de-in e lacing al- go i hms ha e been applied. The Peak Sig- nal o Noise Ra io (PSNR) is used as igu e o me i , o e alua e he quali y be ween he ob ained in e pola ed ames and he o iginal ones. The p oposed algo i hm has been also com- pa ed wi h o he de-in e lacing algo i hms wi h less o simila compu a ional cos : ou spa ial me hod such as line doubling, line a e age, and con en ional ELA (edge- adap i e in e pola ion algo i hm [2]) using 3+3 and 5+5 aps; he simples empo al de-in e lacing algo i hm called ield inse ion, and wo e ico- empo al il e ing wi h wo and h ee ields [2]; and, inally he uzzy mo- ion adap i e algo i hms epo ed in [5] and [6]. Table 1 shows he a e age PSNR alues ob- ained when de-in e lacing i y ields o se en ideo sequences. The PSNR esul s show ha he p oposed algo i hm pe o ms be e han he o he algo i hms since i achie es he highes alues. Mo eo e , i s compu a ional complexi y is qui e low since he h ee uzzy sys ems a e e y simple. The algo i hm has also been es ed o de- in e lace he eal ilm sequences shown in Ta- ble 2. These esul s p o e he ad an ages o he inclusion o he second uzzy sys em (F S2). Finally, he supe io pe o mance o ou ap- p oach can be co obo a ed by he isual in- 1180 P oceedings o IPMU’08 Table 2: A e age PSNR alues (in dBs) using ilm sequences. SEQUENCE Fi e Rose Chop Hun Fa go Repai Fa go Tokyo FORMAT PAL TV PAL TV PAL TV PAL TV PAL TV Line Doubling 34.51 39.97 30.48 28.79 27.22 Line A e age 38.76 44.61 35.92 34.31 31.46 ELA 3+3 35.55 44.07 35.28 33.66 30.02 ELA 5+5 33.61 43.16 34.33 32.16 28.53 Field Inse ion 36.41 24.06 31.23 33.07 36.49 VT 2 ields 40.32 44.18 35.87 40.99 36.84 VT 3 ields 41.16 46.08 38.43 38.91 35.13 Technique in [5] 39.36 43.71 36.64 40.11 34.88 Technique in [6] 41.14 42.64 37.33 42.54 37.71 P oposal 42.11 48.81 41.63 42.81 37.75 spec ion o he de-in e laced ames om he Ca phone sequence shown in Figu e 7. 4 Conclusions The algo i hm p esen ed he ein is he esul o he applica ion o uzzy logic-based sys ems o ideo p ocessing. Especially his app oach ackles he p oblem o de-in e lacing, which is cu en ly mo e demanded in consume de- ices. The algo i hm o e comes he pe o - mance o o he well-known de-in e lacing al- go i hms by adap ing he in e pola ion s a - egy o he p esence o mo ion and edges. To achie e i , he app oach includes h ee uzzy sys ems: one is used o combine a spa ial and a empo al in e pola o acco ding o he le el o mo ion, and he o he wo p o ide a sma empo al and spa ial in e pola o . Acknowledgemen s This wo k has been suppo ed in pa by he Spanish MEC P ojec s TEC2005-04359 and DPI2005-02293, and by he P ojec s TIC2006-635 and TEP2006-375 om he An- dalusian egional Go e nmen . Re e ences [1] J. Whi ake . Tele ision ansmissions sys ems, chap e book o ’S anda d hand- book o ideo and ele ision enginee - ing’. McG aw-Hill Edi o ial, Blacklick OH (USA), 2002. [2] G. de Haan. De-in e lacing, chap e book o ’Digi al Video. Pos P ocessing’, pages 185-201, Uni e si y P ess Eindho en, Sep. 2006. [3] A. M. Bock. Mo ion-adap i e s anda ds con e sion be ween o ma s o simila ield a es. Signal P ocessing: Image Communica ion, ol.6, no.3, pages 275- 280, 1994. [4] H. Jiang, D. Huu and E. Tinyo k. Mo ion adap i e dein e lacing. Uni ed S a es Pa en (US 6,459,455), Oc . 2002. [5] D. Van de Ville, R. Van de Wall, W. Philips and I. Lemahieu. Mo ion adap i e de-in e lacing using uzzy logic. P oc. In e na ional Con e ence on In o ma- ion P ocessing and Managemen o Un- ce ain y in Knowledge-Based Sys ems (IPMU), pages 1989-1996, Jul. 2002. [6] J. Gu i´e ez-R´ıos, F. Fe n´andez- He n´andez, J. C. C espo and G. T e i˜no. Mo ion adap i e uzzy ideo de-in e lacing me hod based on con o- lu ion echniques. In P oceedings o he con e ence IPMU’2004, pages 1635-1642, Pe ugia, I aly, Jul. 2004. [7] P. B ox I. Ba u one S. S´anchez-Solano J. Gu i´e ez-R´ıos and F. Fe n´andez- He n´andez. A uzzy edge-dependen mo ion adap i e algo i hm o de- in e lacing. Fuzzy Se s and Sys ems, ol.158, no.3, pages 337-347, Feb.2007. P oceedings o IPMU’08 1181 LINE DOUBLING LINE AVERAGE ELA 3+3 ELA 5+5 FIELD INSERTION VERTICO TEMPORAL 2 FIELDS VERTICO TEMPORAL 3 FIELDS TECHNIQUE IN [7] VAN DE VILLE ET AL. [6] PROPOSED ALGORITHM Figu e 7: De-in e laced ames o Ca phone sequence. [8] P. B ox I. Ba u one and S. S´anchez- Solano. A uzzy mo ion adap i e algo- i hm o in e laced- o-p og essi e con- e sion. In P oceedings o he Con e ence IPMU’2006, Pa ´ıs, F ance, Jul. 2006. [9] P. B ox I. Ba u one and S. S´anchez- Solano. Fuzzy mo ion adap i e algo- i hm o ideo de-in e lacing. In P o- ceedings o he Con e ence KES’2006, Bou nemou h, Uni ed Kingdom, Oc . 2006. 1182 P oceedings o IPMU’08