Op ical Flow Es ima ion wi h Consis en Spa io- empo al
Cohe ence Models
Ja ie S´
anchez, Agus ´
ın Salgado and Nelson Monz´
on
Cen o de Tecnolog´
ıas de la Imagen (CTIM)
Depa amen o de In o m´
a ica y Sis emas
Uni e si y o Las Palmas de G an Cana ia, Spain
{jsanchez, asalgado}@dis.ulpgc.es, [email p o ec ed]
Keywo ds: Op ical Flow, Va ia ional Me hods, PDE, Tempo al Cohe ence.
Abs ac : In his wo k we p opose a new a ia ional model o he consis en es ima ion o mo ion ields. The aim o
his wo k is o de elop app op ia e spa io- empo al cohe ence models. In his sense, we p opose wo main
con ibu ions: a nonlinea low cons ancy assump ion, simila in spi i o he nonlinea b igh ness cons ancy
assump ion, which con enien ly ela es low ields a di e en ime ins an s; and a nonlinea empo al egula -
iza ion scheme, which complemen s he spa ial egula iza ion and can cope wi h piecewise con inuous mo ion
ields. These con ibu ions pose a cong uen a ia ional model since all he ene gy e ms, excep he spa ial
egula iza ion, a e based on nonlinea wa pings o he low ield. This model is mo e gene al han i s spa ial
coun e pa , p o ides mo e accu a e solu ions and p ese es he con inui y o op ical lows in ime. In he
expe imen al esul s, we show ha he me hod a ains be e esul s and, in pa icula , i conside ably imp o es
he accu acy in he p esence o la ge displacemen s.
1 INTRODUCTION
The es ima ion o mo ion ields is a key p oblem in
compu e ision. I se es as a basis o many appli-
ca ions, such us s e eoscopic ision and 3D scene e-
cons uc ion, medical image analysis, s uc u e om
mo ion, objec acking and o he s. I we a e gi en
a ideo sequence, and we wan o ind he mo ion o
he objec s in he images, ou me hod should p o ide
a solu ion ha is consis en h ough he sequence. In
his wo k, we add ess he p oblem o empo al co-
he ence in op ical low me hods. The aim is o de ise
new me hods ha allow inding con inuous low ields
in ime.
Op ical low me hods can be u he imp o ed i
empo al in o ma ion is p ope ly managed (Weicke
and Schn¨
o , 2001). In his wo k, he au ho s p opose
a me hod ha is a s aigh ex apola ion o he spa ial
cohe ence model o he empo al dimension, based on
a con inuous spa io- empo al egula iza ion scheme.
Mo e ecen ly, some au ho s ha e gene alized he use
o he low empo al de i a i e. Typically, he em-
po al in o ma ion is coupled wi h he spa ial g adien
in he o m o a non-quad a ic 3D smoo hing ope a-
o . Howe e , in (S´
anchez e al., 2012), he au ho s
analyze he beha io o a con inuous empo al egu-
la ize and show se e al expe imen s whe e i ails.
Black (Black, 1994) uses obus unc ionals o
deal wi h ou lie s and in oduces a empo al con inu-
i y s a egy o accoun o he empo al cohe ence o
he sequence. This empo al con inui y is based on a
p edic ion s ep and an a achmen o he low o he
p edic ed alue. I wa ps he low ield o es ima e i s
alue in he ollowing ame. This is in e es ing, be-
cause he wa ping allows inding he co ec low co -
espondences. Mo e ecen ly, he e has been se e al
wo ks dealing wi h empo al cohe ence in di e en
ways: o ins ance, in (Sun e al., 2010) he empo al
consis ency is es ablished easoning on he segmen a-
ion on laye s.
We p opose se e al con ibu ions: on he one
hand, we in oduce a nonlinea low cons ancy as-
sump ion ha i s wi h he nonlinea da a assump-
ion; on he o he hand, we p opose a no el non-
linea low egula iza ion scheme ha can deal wi h
non-con inuous op ical lows. Ano he con ibu ion
is a new anis opic di usion ope a o based on he
Nagel-Enkelmann ope a o . This new ope a o allows
espec ing he objec bounda ies du ing he di usion
p ocess, a he same ime ha i a oids o e segmen a-
ion in ex u e egions.
The o me con ibu ion was mo i a ed by he e-
sul s p esen ed in (Salgado and S´
anchez, 2006). The
expe imen al esul s showed ha he use o a nonlin-
ea empo al o mula ion o he low ield p o ided
e y good esul s. Tha was he i s ime ha such
a nonlinea low assump ion was in oduced. Fo he
second con ibu ion, we in oduce a non-con inuous
low egula iza ion scheme a he PDE le el. This is
a pu e egula iza ion app oach ha eplaces he adi-
ional con inuous empo al smoo hing.
In Sec ion 2 we examine he new ene gy model
and explain he no el empo al cohe ence s a egy.
The minimiza ion o he ene gy model and some nu-
me ical de ails a e explained in Sec ion 3. In he ex-
pe imen al esul s – Sec ion 4 – we es ou me hod
using a syn he ic sequence. Finally he conclusions in
Sec ion 5.
2 NONLINEAR VARIATIONAL
MODEL
I we ha e a se o images Ij(x), wi h j=1, .., N,
N he numbe o ames and x= (x,y), he aim is
o ind a se o op ical low unc ions, {hi(x)}, wi h
i=1, .., N−1. We decompose ou ene gy unc ional
in wo sepa a e pa s:
E({hi(x)}) = ES({hi(x)})+ET({hi(x)}).(1)
The i s e m on he igh , ES, s ands o he spa-
ial ene gy model and he second e m, ET, is he en-
e gy model co esponding o he empo al cohe ence
s a egy. The spa ial model eads as ollows:
ES=ZN−1
∑
i=1
Ψ(Ii(x)−Ii+1(x+hi(x)))2dx
+γZN−1
∑
i=1
Ψk∇Ii(x)−∇Ii+1(x+hi(x))k2dx
+αZN−1
∑
i=1
Ψ(N(∇Ii,∇hi))dx,(2)
wi h Ψs2=√s2+ε2(εa p e ixed small con-
s an , e.g. 0.01). This kind o unc ion mi iga es
he e ec o ou lie s and beha es like TV egu-
la iza ion app oaches when used in he smoo hness
e m. The ad an age o a TV smoo hing scheme
is ha i p ese es discon inui ies o he low. We
use he aniso opic di usion ope a o , N(∇Ii,∇hi) =
ace∇hT
i(x)D(∇Ii)∇hi(x), p oposed in (Nagel
and Enkelmann, 1986), which p ese es discon inu-
i ies o he images in he low ield, D(.)de ined as:
D(∇I) = ∇I⊥T∇I⊥+λ2Id
k∇Ik2+2λ2,
wi h Id he iden i y ma ix. λde e mines he g adien
alue om which he aniso opy is ac i a ed. This
pa ame e can be compu ed om he mo e in ui i e
iso opic ac ion, 0 ≤s≤1, in oduced in ( ´
Al a ez
e al., 2000).
Fo he empo al ene gy model, we ollow he
ideas p esen ed in (Salgado and S´
anchez, 2006).
Gi en ha an objec in he sequence may unde go
la ge displacemen s, we ha e o deal wi h in o ma-
ion ha is wa ped h ough he lows. In ac , gi en a
low hi(x), a ins an i, i s co esponding low in he
ollowing ime ins an is hi+1(x+hi(x)). I hi(x)is
la ge, hen he empo al de i a i e canno be com-
pu ed, bu he p e ious co espondence s ill holds.
Thus, one way o ela e mo ion ields a di e en
ime ins an s is h ough he low cons ancy assump-
ion (FCA), hi(x) = hi+1(x+hi(x)). The e o e, he
empo al cohe ence model, ET, can be o mula ed as,
ET=βZN−2
∑
i=1
Φkhi(x)−hi+1(x+hi(x))k2dx,
(3)
wi h Φs2=e−k∇Ikκ√s2+ε2, wi h κ=0.8 and ε=
0.01.
This e m is cong uen wi h he b igh ness and
g adien cons ancy e ms. In he p esence o la ge
displacemen s, his empo al model is cohe en wi h
he spa ial o mula ion and ela es alues a he co -
ec posi ions. No e ha when objec displacemen s
a e e y small, his e m can be seen as an app oxi-
ma ion o he empo al de i a i e o he low, which
has shown o be e ec i e in a con inuous se ing (e.g.,
(Weicke and Schn¨
o , 2001) o (Papenbe g e al.,
2006)).
3 MINIMIZING THE ENERGY
MODEL
In his sec ion we de i e he Eule -Lag ange equa-
ions o (2) and (3). Then, we in oduce a nonlinea
egula iza ion scheme a he PDE, which closely e-
sembles a con inuous empo al smoo hing app oach.
The Eule -Lag ange equa ions o he spa ial en-
e gy model (2) a e:
0=Ψ0(Ii(x)−Ii+1(x+hi(x)))2
·(Ii(x)−Ii+1(x+hi(x)))·∇Ii+1(x+hi(x))
+γ Ψ0k∇Ii(x)−∇Ii+1(x+hi(x))k2
·(∇Ii(x)−∇Ii+1(x+hi(x)))·HIi+1(x+hi(x))
+αdi Ψ0(N(∇Ii,∇hi))·D(∇Ii)·∇hi,(4)
whe e HIi+1is he Hessian ma ix. The empo al en-
e gy model (3) yields he ollowing Eule -Lag ange
equa ions:
0=β Φ0khi(x)−hi+1(x+hi(x))k2
·(hi(x)−hi+1(x+hi(x)))T
·Id −∇hT
i+1(x+hi(x))
+β Φ0
hi(x)−hi−1(x+h∗
i−1(x))

2
·hi(x)−hi−1(x+h∗
i−1(x))·|J(x)|,(5)
whe e |J(x)|s ands o he absolu e alue o he Jaco-
bian ma ix, wi h J(x) = 1+u∗
i−1,x1+ ∗
i−1,y−
u∗
i−1,y ∗
i−1,x.h∗
i−1=u∗
i−1, ∗
i−1Tis he backwa d low
om ame Ii o Ii−1.
In o de o de i e (hi(x)−hi+1(x+hi(x))) wi h
espec o hi+1(x), we can use he change o a iables
z=x+hi−1(x). This change allows us o emo e he
nonlinea i y inside he low. The backwa d low, h∗
i−1,
na u ally appea s due o his change o a iables.
We use a g adien descen app oach o ind he so-
lu ion o he abo e PDE. The nonlinea e ms, e.g.
Ii+1(x+hi(x)), a e linea ized using i s o de Tay-
lo expansions. In he empo al cohe en amewo k,
we use Di ichle bounda y condi ions o he las and
i s ames, whe eas Neumann bounda y condi ions
a e used in he spa ial domain. We use a s anda d
coa se- o- ine s a egy o deal wi h la ge displace-
men s, based on a py amidal s uc u e. The sys em
o equa ions is spa se, so i can be e icien ly sol ed
by means o he Gauss-Seidel o SOR me hod in each
scale.
We in oduce a nonlinea empo al smoo hing
scheme. I s o mula ion is in ui i ely de i ed om
he second o de empo al de i a i e o he low ield,
u ≈ui,j,k+1−2ui,j,k+ui,j,k−1. In he PDE, his sec-
ond o de de i a i e has a con inuous empo al egu-
la izing e ec ha is consis en i he low ield a ies
smoo hly ac oss he image sequence. We p opose a
new solu ion, which is simila in spi i o his nume -
ical app oxima ion, and is sui able o dealing wi h
non-con inuous displacemen s. This is a nonlinea
o mula ion ha pu s in o co espondence he co ec
low alues in di e en ames. I is no e iden how
o abs ac his idea a he ene gy le el in Equa ion
(3). As be o e, we also use L1 unc ions o u n he
me hod mo e obus agains ou lie s, in he ollowing
way:
TS=δ Φ0
hi−1(x+h∗
i−1(x))−hi+1(x+hi(x))

2
·hi−1(x+h∗
i−1(x))−2hi(x)+hi+1(x+hi(x))
(6)
This e m p o ides a new scheme a he PDE le el
and has o be combined wi h he p e ious PDE equa-
ions (4) and (5). In he expe imen s, we show ha
his nonlinea smoo hing p o ides e y good esul s:
i has a simila gain as in he con inuous case, bu i
co ec ly handles la ge discon inui ies in he mo ion
ield.
4 EXPERIMENTAL RESULTS
Nex we examine he beha io o he empo al mod-
els in oduced in equa ions (1) and (6). Fo his, we
use a simple sequence o a squa e ansla ing o e a
ex u ed backg ound. The squa e is mo ing 15 pixels
pe ame, while he backg ound mo es 3 pixels in he
same di ec ion. In he i s ow o Fig. 1, we show he
hi d ame o he squa e sequence, i s g ound u h,
and he bes spa ial solu ion ound. In he second ow,
we show h ee empo al solu ions: he i s o he
nonlinea empo al a achmen de ined in (3); he sec-
ond, o he nonlinea empo al smoo hnes app oach
de ined in (6); and, inally, using bo h empo al e ms.
The colo , in he mo ion ield, ep esen s he di ec ion
and, he in ensi y, i s magni ude.
Figu e 1: Squa e sequence. Fi s ow: one o he images o
he Squa e sequence, he g ound u h and he bes spa ial
solu ion ound. Second ow: h ee empo al solu ions wi h
β=8, δ=25 and (β=1,δ=25), espec i ely.
The imp o emen o he empo al me hods wi h
espec o he spa ial solu ion is impo an . As ex-
pec ed, he spa ial me hod p oduces highe e o s a
he mo ion discon inui ies and, mo e signi ican ly, a
he occlusions. Table 1 shows he a e age End-poin
(EPE) and Angula (AAE) e o s o hese esul s.
The i s empo al esul , co esponding o he i s im-
age in he second ow o Fig. 1, p o ides an impo an
imp o emen on he EPE and, mo e no iceable, on he
AAE. The imp o emen in accu acy is s ill mo e im-
po an i we use he nonlinea empo al smoo hing
scheme (Equa ion (6)) o a combina ion o bo h.
We obse e ha he nonlinea empo al smoo h-
ing scheme (6) beha es be e han he empo al a -
achmen , e en a he mo ion bounda ies. The g aph-
ics in Fig. 2 show he EPE o e e y ame on he
Table 1: EPE and AAE o he Squa e sequence.
Me hod EPE AAE
Spa ial 0.071 0.629o
Tempo al 1 (β=8) 0.049 0.204o
Tempo al 2 (δ=25) 0.036 0.134o
Tempo al 3 (β=1,δ=25) 0.035 0.138o
squa e sequence. F ame by ame, he op ical lows
a e mo e accu a e in he empo al me hods. We also
obse e ha he esul s a e e y s able, especially in
he middle o he ’Tempo al 2 (δ)’ line. Reasonably,
he ames a he beginning and end o he sequence
p esen highe e o s, due o he Di ichle bounda y
condi ions.
Figu e 2: EPE in each op ical low o he Squa e sequence.
5 CONCLUSIONS
In his pape we ha e p esen ed a new spa io- empo al
cohe ence model o he consis en es ima ion o op-
ical lows. We ha e ocused on di e en nonlinea
low assump ions ha a e mo e con iden in he es-
ima ion o mo ion ields han p e ious app oaches.
These nonlinea assump ions co ec ly i wi h he
s anda d nonlinea b igh ness and g adien cons ancy
e ms, can cope wi h gene al image sequences and
p o ide be e solu ions. In pa icula , we ha e p o-
posed wo main con ibu ions: on he one hand, we
ha e in oduced he nonlinea low cons ancy assump-
ion (FCA) in he ene gy model. This e m ela es
low ields a di e en ime ins an s and is consis-
en wi h he es o he ene gy e ms. On he o he
hand, we ha e p oposed a nonlinea empo al di u-
sion scheme a he PDE le el, which p oduces con-
inuous lows in ime. We ha e seen ha his new
scheme is mo e gene al han using he con inuous
empo al egula iza ion o he low, wi h he ad an-
age ha i con enien ly deals wi h con inuous and
non-con inuous eloci ies. In ac , i he mo ion is
e y small, his e m app oxima es a con inuous em-
po al smoo hing scheme. In he expe imen al esul s,
we ha e shown ha he me hod p o ides impo an
accu acy imp o emen s, specially in he p esence o
la ge displacemen s. The esul s a e p omising in bo h
cases, al hough we obse e a be e pe o mance o
he nonlinea empo al smoo hing scheme in gene al.
Ano he in e es ing esul o he empo al cohe ence
schemes is ha he backg ound mo ion oscilla ions
end o disappe . These oscilla ions clea ly appea in
he spa ial me hod, in egions whe e he e is no appa -
en mo ion.
ACKNOWLEDGEMENTS
This wo k has been pa ly ounded by he Spanish
Minis y o Science and Inno a ion h ough he e-
sea ch p ojec TIN2011-25488.
REFERENCES
´
Al a ez, L., Weicke , J., and S´
anchez, J. (2000). Reli-
able es ima ion o dense op ical low ields wi h la ge
displacemen s. In e na ional Jou nal o Compu e Vi-
sion, 39(1):41–56.
Black, M. J. (1994). Recu si e non-linea es ima ion o dis-
con inuous low ields. In P oceedings o he hi d Eu-
opean con e ence on Compu e ision ( ol. 1), ECCV
’94, pages 138–145, Secaucus, NJ, USA. Sp inge -
Ve lag New Yo k, Inc.
Nagel, H. H. and Enkelmann, W. (1986). An in es iga ion
o smoo hness cons ain s o he es ima ion o dis-
placemen ec o ields om image sequences. IEEE
T ansanc ions on Pa e n Analysis and Machine In el-
ligence, 8:565–593.
Papenbe g, N., B uhn, A., B ox, T., Didas, S., and Weick-
e , J. (2006). Highly Accu a e Op ic Flow Compu-
a ion wi h Theo e ically Jus i ied Wa ping. In e na-
ional Jou nal o Compu e Vision, 67(2):141–158.
Salgado, A. and S´
anchez, J. (2006). A empo al egula ize
o la ge op ical low es ima ion. In IEEE In e na-
ional Con e ence on Image P ocessing ICIP, pages
1233–1236.
S´
anchez, J., Monz´
on, N., and Salgado, A. (2012). Robus
op ical low es ima ion. IPOL: Image P ocessing On-
line, P ep in :1–18.
Sun, D., Sudde h, E., and Black, M. J. (2010). Lay-
e ed Image Mo ion wi h Explici Occlusions, Tempo-
al Consis ency, and Dep h O de ing. In La e y, J.,
Williams, C. K. I., Shawe-Taylo , J., Zemel, R., and
Culo a, A., edi o s, Ad ances in Neu al In o ma ion
P ocessing Sys ems 23, olume 23, pages 2226–2234.
Weicke , J. and Schn¨
o , C. (2001). Va ia ional Op ic
Flow Compu a ion wi h a Spa io-Tempo al Smoo h-
ness Cons ain . Jou nal o Ma hema ical Imaging
and Vision, 14(3):245–255.