Recei ed 21 Augus 2024, accep ed 7 Oc obe 2024, da e o publica ion 11 Oc obe 2024, da e o cu en e sion 24 Oc obe 2024.
Digi al Objec Iden i ie 10.1109/ACCESS.2024.3478793
Pe o mance E alua ion o Objec i e Quali y
Assessmen Me hods o Omnidi ec ional
Images Unde Eme ging Comp essions
MAREK SIMKA 1, LADISLAV POLAK 1, (Membe , IEEE), MARTIN NOVOTNY 2, JAN KUFA 1,
AND KAREL FLIEGEL 2, (Membe , IEEE)
1Depa men o Radioelec onics, Facul y o Elec ical Enginee ing and Communica ion, B no Uni e si y o Technology, 61600 B no, Czech Republic
2Depa men o Radioelec onics, Facul y o Elec ical Enginee ing, Czech Technical Uni e si y in P ague, 166 27 P ague, Czech Republic
Co esponding au ho : Ma ek Simka ([email p o ec ed])
This wo k was suppo ed in pa by he In e nal G an Agency o B no Uni e si y o Technology unde P ojec FEKT-S-23-8191; in pa by
he G an Agency o he Czech Technical Uni e si y in P ague unde P ojec SGS23/185/OHK3/3T/13 and P ojec
SGS23/186/OHK3/3T/13; and in pa by he Minis y o Educa ion, You h and Spo s o he Czech Republic, unde P ojec LTT20004.
ABSTRACT This pape p esen s a unique s udy o objec i e and subjec i e image quali y assessmen
(IQA) o 360◦images, dis o ed by eme ging and legacy comp essions - AV1 Image File Fo ma (AVIF),
Join Pho og aphic Expe s G oup (JPEG), High E iciency Image File Fo ma (HEIC), JPEG XL om he
c oss-lab expe imen s ca ied ou by wo uni e si ies. The pe o mance e alua ion o objec i e con en ional
IQA me hods elied on s a is ical da a analysis employing no only adi ional app oaches wi h co ela ion
coe icien s bu also ad anced echniques such as Recei e Ope a ing Cha ac e is ic (ROC) analysis, which
o e s se e al s a is ical ad an ages. The comp ehensi e s a is ical examina ion p esen ed, u ilizing an
ou come om he subjec i e expe imen wi h 66 obse e s, con i med ha he pe o mance o speci ic
IQA me ics depends no only on he image ea u es bu also on he applied image comp ession algo i hm.
O e all, he s udy examined nine IQA objec i e me ics. Fo ins ance, when e alua ing dis o ion caused
by he AVIF and HEIC codecs, he Mul i-Scale S uc u al Simila i y (MS-SSIM) me ic exhibi ed he bes
pe o mance, while he Fea u e Simila i y ch ominance (FSIMc) me ic excelled in assessing JPEG-based
comp essions. Addi ionally, de ailed ou comes o he subjec i e expe imen s, including eye- acking da a,
a e a ailable in a public da ase . This s udy p o ides p e iously unexplo ed insigh s in o he pe o mance o
con en ional objec i e me ics o 360◦images deg aded by eme ging image codecs. The p esen ed esul s
sugges sui able me ics o speci ic applica ion scena ios (codecs) and, along wi h he da ase , lay he
g oundwo k o u he esea ch on new objec i e quali y assessmen me hods o omnidi ec ional images
unde eme ging comp essions.
INDEX TERMS Omnidi ec ional image (360◦), 360-deg ee con en , i ual eali y, objec i e quali y
assessmen , subjec i e es , pe cep ual image quali y.
I. INTRODUCTION
The p oli e a ion o imme si e echnologies such as i ual
and augmen ed eali y (VR and AR) has escala ed he
need o supe io omnidi ec ional con en . Omnidi ec ional
images, simply called 360◦images, p o ide use s wi h a ull
The associa e edi o coo dina ing he e iew o his manusc ip and
app o ing i o publica ion was And ea F. Aba e .
360◦pano amic iew o a scene, deli e ing an imme si e
and cap i a ing isual expe ience. Recen ly, subs an ial ile
sizes o omnidi ec ional images p esen hu dles in s o age,
ansmission, and ende ing. To add ess hese challenges
e ec i ely, i is c ucial o u ilize e icien comp ession
echniques ha p o ide high lexibili y while p ese ing
he pe cep ual quali y o he comp essed 360◦images [1].
Mo eo e , o assess he equi ed Quali y o Expe ience (QoE)
VOLUME 12, 2024
2024 The Au ho s. This wo k is licensed unde a C ea i e Commons A ibu ion-NonComme cial-NoDe i a i es 4.0 License.
Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by-nc-nd/4.0/ 150419
M. Simka e al.: Pe o mance E alua ion o Objec i e Quali y Assessmen Me hods
o human obse e s, eliable me hods able o de e mine he
isual quali y o omnidi ec ional images a e i al [2],[3].
Nowadays, he assessmen o he 360◦images isual
quali y has become a p ominen esea ch a ea wi hin he
image quali y assessmen (IQA) domain [4]. Objec i e
me ics, which ely on ma hema ical models, a e designed o
gauge he pe cep ual quali y o comp essed images. Howe e ,
mos me ics we e designed and uned o con en ional
2D images, so he e emains a gap o esea ch on he
pe o mance o hese me hods o 360◦images [5] o
de elopmen o no el me hods uned o his ype o con en s.
These me ics ake in o accoun ac o s such as spa ial and
angula dependencies, s uc u al simila i y, colo accu acy,
ex u e p ese a ion, and isual ideli y. Con e sely, IQA
h ough subjec i e means in ol es conduc ing es s in a
con olled labo a o y en i onmen , whe e human obse e s
p o ide hei subjec i e e alua ions o he pe cei ed quali y
o dis o ed images when iewed wi h a head-moun ed
display (HMD) [6].
In ecen yea s, subs an ial p og ess has been made in he
ield o IQA due o esea ch e o s [7]. One ongoing a ea
o esea ch ocuses on de eloping app op ia e me hodologies
o assessing he QoE o VR-based con en . The ele ance
o his esea ch is e idenced by nume ous pape s add essing
IQA in he con ex o 360◦ isual con en , using ei he
objec i e [8] o subjec i e [9],[10] me hods. The impo ance
o e alua ing he pe cep ual quali y o 360◦images ex ends
beyond echnical conside a ions. I has a di ec impac on
he o e all use QoE in VR-based applica ions. By com-
p ehending he pe cep ual ac o s ha in luence image
quali y and le e aging bo h objec i e and subjec i e me hods,
i becomes possible o op imize comp ession algo i hms and
ansmission echniques. This op imiza ion esul s in he
deli e y o high-quali y 360◦images while minimizing he
p esence o no iceable deg ada ions.
A. RELATED WORKS
The impo ance o examining he pe o mance o omnidi-
ec ional IQA (OIQA) h ough bo h objec i e and subjec i e
me hods is e iden om he conside able amoun o exis ing
esea ch. A comp ehensi e o e iew o he ecen s udies
de o ed o IQA (including subjec i e es s) o 2D omnidi ec-
ional images can be ound in Table 1.
A s udy [11] in oduced a es bed o conduc ing subjec i e
assessmen s o omnidi ec ional isual con en . This es bed
collec ed Mean Opinion Sco e (MOS) a ings, iewing
di ec ion da a, and ime spen by up o 48 pa icipan s
using he Absolu e Ca ego y Ra ing wi h Hidden Re e ence
(ACR-HR) me hod. Hal e alua ed equi ec angula o ma ,
and hal assessed cubic p ojec ion images. The MOS esul s
a o ed he equi ec angula o ma o supe io image quali y.
Upenik and e al. in [5] p o ided a s udy compa ing wo
ypes o objec i e ull- e e ence (FR) me ics - adi ional
o 2D con en and specialized me ics o 360◦images.
They comp essed ou s ill images om a ideo da ase by
using Join Pho og aphic Expe s G oup (JPEG), JPEG2000.
and High E iciency Video Coding (HEVC) codecs. While
hey p ima ily used OIQA me ics based on Peak Signal- o-
Noise Ra io (PSNR), me ics IQA like S uc u al Simila i y
Index Measu e (SSIM), Mul i-Scale S uc u al Simila i y
(MS-SSIM), and Visual In o ma ion Fideli y in he pixel
domain (VIFp) consis en ly yielded be e quali y a ings.
No ably, VIFpou pe o med heo he s, wi ha Pea sonLinea
Co ela ion Coe icien (PLCC) alue o 0.8994.
In he wo k [12], he esea che s add ess he challenge o
quali y assessmen o 360◦images. They c ea ed an OIQA
da abase wi h 16 sou ce images and 320 deg aded images,
comp essed using JPEG and JPEG2000. They compa ed nine
FR-IQA me ics. The s udy highligh s he impo ance o
image de ails in he VR expe ience and sugges s using isual
saliency o OIQA. The disad an age is only 20 subjec s
based on he legacy ITU-R BT.500-11 ecommenda ion
designed o TV images, which is no s a is ically su icien
o VR con en .
Au ho s in [13] ga he ed he la ges 360-deg ee image
da ase o da e, known as CVIQD2018, wi h 528 comp essed
images using common coding s anda ds. The goal o he
esea ch was o es i exis ing OIQA me hods could e alua e
comp essed omnidi ec ional image quali y. They used i e
no- e e ence (NR) IQA me ics, en adi ional FR-IQA
me ics, and h ee PSNR-based me ics o 360◦images.
To ensu e usabili y, hey conduc ed subjec i e es ing wi h
20 subjec s using he single-s imulus (SS) me hod. Among
he 13 FR-IQA me ics used, SSIM, In o ma ion con en
weigh ed SSIM (IW-SSIM), and Visual Saliency-Induced
Index (VSI) achie ed he highes consensus wi h subjec i e
sco es. The s udy concludes ha ce ain objec i e me ics o
2D images a e s ill alid and obus o 360◦images.
In [14], simila o [13], he au ho s used he same
image da abase and conside ed p io subjec i e es esul s.
Addi ionally, hey explo ed a mul i-channel con olu ional
neu al ne wo k (CNN) o no- e e ence quali y assessmen o
360◦images. This model pe o med simila ly o FR me ics
and ou pe o med all me ics in he CVIQD2018 da abase
o AVC (as H.264) and JPEG comp ession dis o ion.
Howe e , he s udy lacked basic in o ma ion abou he
es ed sequences (selec ed ypes and amoun s) and elied on
sco es om only 20 subjec s acco ding o ITU-R BT.500-11
ecommenda ion.
A new pe cep ual quali y me ic o VR-based con en ,
e med VA-PSNR, was p oposed in [15]. I in eg a es
PSNR wi h a saliency map de i ed om eal head mo e-
men da a, add essing iewe isual a en ion. This me ic
ep esen s an ad ancemen o e he Weigh ed Sphe ical
PSNR (WS-PSNR) me ic o 360◦con en . By conside ing
iewe in e ac ions and isual a en ion, i acknowledges key
ac o s in omnidi ec ional con en pe cep ion. The me ic was
es ed on images also dis o ed wi h he HEVC In a ame
(HEVC-I) codec as a a ian o comp ess s ill images.
Howe e , one limi a ion is he eliance on empi ical isual
da a, which may no always be eadily accessible. Despi e
150420 VOLUME 12, 2024
M. Simka e al.: Pe o mance E alua ion o Objec i e Quali y Assessmen Me hods
TABLE 1. Su ey o he exis ing s udies on IQA (objec i e & subjec i e) o 2D omnidi ec ional images.
his, he s udy o e s a p omising a enue o e alua ing
omnidi ec ional con en quali y, wi h u he esea ch needed
o alida e i s applicabili y in di e se scena ios and con en
ypes.
In [16], a no el deep lea ning app oach o OIQA, called
DeepVR-IQA, was in oduced. The model demons a es
supe io pe o mance, accu a ely es ima ing pixel quali y and
deli e ing an excellen o e all assessmen . Achie ing his
high pe o mance necessi a ed aining on subjec i e a ings
o up o 720 s imulus, esul ing in a s ong co ela ion wi h
subjec i e sco es, e ec i ely cap u ing image cha ac e is ics
a ec ing isual quali y. A d awback o his me hod is i s
demand o a subs an ial olume o aining da a, which could
be challenging in ce ain applica ions.
The pape [17] examines he impac o display cha ac-
e is ics and human a en ion on 360◦images, c ea ing a
da ase ha includes subjec i e e alua ions o hese images.
The au ho s p opose he so-called CPBQA amewo k,
which combines he Blind/Re e enceless Image Spa ial
Quali y E alua o (BRISQUE) and he In eg a ed Local
Na u al Image Quali y E alua o (IL-NIQE). This amewo k
in eg a es global and local quali y ea u es o cubemaps
o blind quali y assessmen . Fu u e esea ch in his a ea
will ocus on de eloping mo e e icien ea u e ex ac ion
me hods and explo ing he e ec o ho spo maps on image
quali y.
The pu pose o wo k [18] was o unde s and human pe -
cep ion o 360◦image quali y. Subjec i e a ings and iewing
beha io da a unde di e se condi ions we e collec ed,
ocusing on he co ela ion be ween iewing condi ions, non-
uni o m dis o ions, and subjec i e sco es. They in oduced
a no el OIQA model based on a ResNe -50 CNN, inco po-
a ing mul i-scale ea u e ex ac ion and quali y p edic ion
modules. In compa ison wi h o he me ics, hei p oposed
model ou pe o med in mos cases, excep o scena ios
in ol ing Gaussian noise deg ada ion.
Au ho s in [19] and [20] es ablished a new model o
blind/NR OIQA (BOIQA). A colo desc ip o ha e lec s
colo in o ma ion was used. O e all, he model should exhibi
a g adien e lec ion as global s uc u al ea u es and g ay
le els wi h local s uc u al ea u es. They did no ga he hei
own subjec i e a ings bu es ed he model on [12],[13] da a.
The BOIQA wi h suppo ec o eg ession is p omising,
bu he e is scope o mo e alida ion wi h a mo e obus
model [22].
Duan and e al. in he s udy [21] ocused on OIQA o
speci ic s i ching- ela ed in luences. The aw 12 images we e
co up ed du ing he s i ching p ocess namely colo , geome -
ic, blu , and ghos ing dis o ion. Following hese dis o ions
in e wined wi h he acquisi ion o omnidi ec ional con en
and subjec i e a ings by 20 esponden s, hey p oposed a
model sepa a ely o FR and NR OIQA. The impac o
comp ession was no conside ed.
B. ORIGINAL CONTRIBUTIONS
F om he abo e elabo a ed s a e-o - he-a (SoA), i is
possible o de i e he ollowing conclusions. Recen ly, he
OIQA-based s udies s ill do no conside eme ging image
comp ession algo i hms and ela ed speci ic comp ession
a i ac s, namely JPEG XL [23], High E iciency Image File
Fo ma (HEIC) [24] and AV1 Image File Fo ma (AVIF)
[25]. Nex , an analysis o con en ional (non-CNN models)
objec i e me ics o OIQA wi h ex ensi e subjec i e es s
has no been p o ided so a . In pa icula , ho ough s a is ical
pe o mance e alua ion beyond co ela ion coe icien s is
also lacking. Conce ning all he a o emen ioned gaps in he
p io a , he main con ibu ions o ou wo k a e summa ized
as ollows:
•Applicabili y alida ion o eme ging image comp ession
algo i hms (HEIC, JPEG XL, AVIF) o 360◦images.
•E alua ion o objec i e con en ional IQA me hods in
e ms o 360◦images co up ed by eme ging codecs.
VOLUME 12, 2024 150421
M. Simka e al.: Pe o mance E alua ion o Objec i e Quali y Assessmen Me hods
•Ex ensi e subjec i e expe imen s conduc ed by wo lab-
o a o ies, coupled wi h s a is ical analysis by Recei e
Ope a ing Cha ac e is ic (ROC) me hodology, ep esen
an ad anced app oach o pe o mance e alua ion.
•In luence e alua ion o he HMDs and es condi ions on
he QoE by c oss-lab s udy (in e -lab co ela ion [26]).
•P o iding new anno a ed da ase ,1including MOS
alues, aw sco es, and eye- acking da a, as a igo ous
ex ension o ou p e ious wo k [27].
To he bes o ou knowledge, a simila s udy wi h he
de ined objec i es has no been p o ided so a .
C. ORGANIZATION OF THE PAPER
The s uc u e o he pape is ou lined as ollows. In oduc ion
o he s udy p o ided in his pape , he elabo a ion o he
SoA, and he main con ibu ions o his wo k a e included
in Sec ion I. The cha ac e is ics o e e ence 360◦images,
he codecs con igu a ion, and objec i e quali y assessmen
a e p esen ed in Sec ion II. The comple e subjec i e es
me hodology is desc ibed in Sec ion III. The main esul s o
bo h objec i e and subjec i e s udies a e gi en in Sec ion IV.
Finally, Sec ion Vconcludes he pape .
II. OBJECTIVE QUALITY ASSESSMENT OF 360◦IMAGES
This sec ion p o ides de ails on he e e ence images, he
image codecs, and hei con igu a ions and speci ica ions o
he objec i e quali y me ics applied.
A. OMNIDIRECTIONAL IMAGE DATA
In his s udy, i e ca e ully selec ed e e ence 360◦images
and hei dis o ed a ian s we e used om he open da abase
OMNIQAD [28]. A p e iew o hese images is depic ed in
Fig. 1. The Sigh s image was used only in he aining session
o he subjec i e es . The de ailed p ope ies o he e e ence
images, including esolu ion and spa ial in o ma ion (SI)
[29], a e a ailable in Table 2. All hese 360◦images we e
selec ed ollowing expe iewing sessions wi h di e en
ea u es, esul ing in di e se esponses om he used image
codecs. Acco ding o he a ge bi pe pixel (bpp), see
Table 3, a o al o 100 images was e alua ed, as he e
we e 19 dis o ed a ian s o each e e ence image, and he
e e ence image was also included.
B. CODING CONFIGURATIONS
As men ioned abo e, comp ession was pe o med wi h
one legacy s ill image codec (JPEG) and h ee eme ging
comp ession algo i hms, namely: JPEG XL, HEIC, and
AVIF. The JPEG XL is he mos ecen successo o JPEG
wi h highe lexibili y and comp ession e iciency. HEIC
and AVIF, based on licensed HEVC and license- ee AV1
ideo codecs espec i ely, a e eme ging image o ma s. I was
shown ha o con en ional 2D images, JPEG XL, HEIC, and
1h ps://zenodo.o g/doi/10.5281/zenodo.7607070
2h ps://gi hub.com/Telecommunica ion-Telemedia-Assessmen /SITI
FIGURE 1. Snapsho s o he e e ence 360◦images (*Image used only in
he aining session o subjec i e p ocedu e).
TABLE 2. Speci ica ions o he e e ence omnidi ec ional images.
TABLE 3. Desc ip ion o he selec ed a ian s o comp essed images.
AVIF excel in achie ing small ile sizes while p ese ing he
high isual quali y o he images [23],[24],[25].
As a ep esen a i e o abled JPEG-based comp ession,
he JPEG XT [30] e e ence so wa e3has been used. Image
comp ession wi h he AVIF and JPEG XL codecs has been
ealized using he liba codec and libjxl lib a ies in FFmpeg.4
In he case o he HEIC codec, he GIMP52.10.32 g aphics
edi o was used. I mus be no ed ha he selec ed imple-
men a ions o hese image comp ession s anda ds ha e been
used o ob ain ep esen a i e image deg ada ion, and no o
e alua e he ac ual codec pe o mance. The con igu a ions
and se ings o he encode s we e mainly ela ed o de ining
he inpu image, he coding lib a y, and he quan iza ion
pa ame e . Basically, i was a de aul se ing o each
implemen a ion, hus o he pa ame e s emained unchanged.
3h ps://jpeg.o g/jpegx /so wa e.h ml
4h ps://www. mpeg.o g/
5h ps://www.gimp.o g/
150422 VOLUME 12, 2024
M. Simka e al.: Pe o mance E alua ion o Objec i e Quali y Assessmen Me hods
C. OBJECTIVE QUALITY METRICS
In his s udy, we u ilized a ious objec i e quali y me ics
o quan i y he dis o ion in he examined 360◦images. Key
among hese we e he e e ence me ics, speci ically FR-
IQA me ics, including PSNR, SSIM, MS-SSIM, Fea u e
Simila i y ch ominance (FSIMc), G adien Magni ude Sim-
ila i y De ia ion (GMSD), VIFp, and Video Mul i-Me hod
Assessmen Fusion (VMAF). The PSNR, SSIM [31], and
VMAF6sco es we e ob ained by FFmpeg amewo k using
he la i and lib ma lib a ies, espec i ely. The MS-SSIM
[32] and VIFp [33] me ics we e compu ed in MATLAB
using an a ailable sc ip .7The FSIMc me ic was used
acco ding o [34], and GMSD was u ilized in i s o iginal o m
as in oduced in [35].
The WS-PSNR [36] and C as e Pa abolic P ojec ion
PSNR (CPP-PSNR) [37] we e chosen as special FR-OIQA
me ics ailo ed o equi ec angula p ojec ion con en . These
objec i e me hods ha e been applied by he command ool.8
The selec ion o he applied objec i e IQA me hods is
based on he mos commonly used con en ional (non-CNN
models) me ics men ioned in he ele an p io a [5],
[11],[12],[13],[14],[15],[16],[17],[18],[19],[20],
[21] dealing wi h 360◦image quali y (see in Subsec ion I-A).
This will es ablish he applicabili y and pe o mance o hese
me ics e en o images comp essed wi h eme ging codecs.
III. SUBJECTIVE QUALITY ASSESSMENT OF 360◦IMAGES
This sec ion desc ibes he subjec i e es p ocedu e, including
de ails o he es condi ions, equipmen , and subjec s.
A. EXPERIMENTAL ENVIRONMENT AND SETUP
The subjec i e es s we e conduc ed in wo labo a o ies
wi h con olled iewing condi ions. One is he Depa men
o Radioelec onics (DREL) a he B no Uni e si y o
Technology (BUT), while he o he is DREL a he Czech
Technical Uni e si y in P ague (CTU). Among o he s, such
a c oss-lab s udy acili a es a mo e complex e alua ion
o a ious ac o s, such as HMD used, on he o e all use s’
QoE.
In he subjec i e expe imen s ealized a he BUT, he
Oculus Ri S9HMD was u ilized as a VR headse . I ea u es
an LCD panel wi h a esolu ion o 1280×1440 (pe eye),
a e esh a e o 80Hz, and a Field o View (FoV) o 88◦. The
headse was connec ed o a lap op unning Oculus so wa e.
To ensu e com o able iewing o images om a ious
di ec ions, obse e s we e p o ided wi h swi el chai s.
A he CTU he HTC Vi e P o Eye10 HMD was used. This
HMD has an OLED display wi h a esolu ion o 1440×1600
(pe eye), a e esh a e o 90Hz, and a isible FoV o 110◦.
A headse is also equipped wi h eye- acking senso s. The
6h ps://gi hub.com/Ne lix/ ma
7h ps://gi hub.com/sa a ab/image-quali y- ools
8h ps://gi hub.com/Samsung/360 ools
9h ps:// -compa e.com/headse /oculus i s
10h ps://www. i e.com/au/p oduc / i e-p o-eye/specs/
used HMD was connec ed o a PC (HP Z230 wo ks a ion wi h
RTX 2060 GPU) unning a es applica ion.
B. SUBJECTIVE EXPERIMENT PROCEDURE
Following heITU-T P.919[38] ecommenda ion,we adop ed
he ACR-HR me hodology wi h a i e-le el a ing scale
(1-bad, 2-poo , 3- ai , 4-good, and 5-excellen ) o IQA o
360◦images. In con as o con en ional subjec i e expe -
imen s (wi h PC moni o , TV display, e c.), en i onmen al
ac o s such as iewing dis ance and ligh ing condi ions a e
no equi ed o be conside ed. The expe imen p ocedu e,
in bo h labs, consis s o wo pa s - aining and es sessions.
1) BUT
The aining session aimed o in oduce he subjec i e
expe imen and amilia ize he pa icipan s wi h he VR
echnology and omnidi ec ional images. The es p inciple
and he pu pose o he expe imen we e explained ini ially.
Subsequen ly, a ques ionnai e ga he ed basic in o ma ion
abou pa icipan s (age, gende , wea ing glasses, e c.).
Nex , a ision sc eening assessed isual acui y using he
Snellen cha and colo sensi i i y h ough he Ishiha a
es . A e amilia izing wi h he HMD, each pa icipan
iewed i e sample images showcasing 360◦con en ,
speci ically comp essed a ian s o he Sigh s image (see
Fig. 1* )), se ing as aining examples. This p ocess helped
pa icipan s unde s and how o p o ide pe cep ual quali y
a ings.
The aining session is ollowed by he es one. I s
imeline is depic ed in Fig. 2. Each es s imulus ( es ed
image), including he e e ence, was p esen ed one a a
ime in a pseudo- andom o de . Obse e s had a maximum
iewing ime o 15seconds o less, depending on he
obse e ’s decision. Subsequen ly, a o ing pe iod was ini i-
a ed, displaying a a ing scale on a g ay backg ound. Du ing
his ime, subjec s e bally communica ed hei subjec i e
sco es, which we e eco ded by he expe imen e on a PC.
Sequence ansi ions we e au oma ed a e 15 seconds o
manually ad anced by he expe imen e using a con olle ,
ensu ing consis en condi ions o each subjec . On a e age,
pa icipan s ook 19minu es o comple e he subjec i e
expe imen es session. Obse e s had he op ion o expedi e
he p ocess by submi ing hei quali y sco e be o e he
ime limi . Mo eo e , subjec s could eques a pause a
any ime o a b eak and es , e ec i ely a oiding iewing
a igue.
FIGURE 2. The imeline o he subjec i e es wi h he ACR-HR me hod
(TS1 ep esen s he i s es ed s imulus, TS2 he second es ed s imulus).
Du ing he o ing in e al, a g ay backg ound wi h a a ing scale was
displayed.
VOLUME 12, 2024 150423

M. Simka e al.: Pe o mance E alua ion o Objec i e Quali y Assessmen Me hods
2) CTU
As a he BUT, he es was di ided in o wo pa s.
Du ing he aining session, he subjec s we e sea ed (a
s a iona y chai was used a he CTU), in oduced o
he es p inciple, and amilia ized wi h he con ols o
he subjec i e expe imen applica ion. Subsequen ly, he
subjec s illed ou he ques ionnai e ga he ing hei pe sonal
in o ma ion (iden ical o he BUT). In opposi ion o he BUT,
no ision sc eenings we e held wi h he subjec s du ing he
aining session a he CTU, and he subjec s we e asked
o epo e en ual ision de iciencies i known. Subjec s
wi h diagnosed ision de iciencies we e no included in he
s udy.
The en i e es session was au oma ed wi h an applica ion
de eloped a he CTU using Uni y Engine.11 Each pa icipan
was shown comp essed 360◦con en (also wi h hidden
e e ence) in andomized o de and one a a ime. The display
ime o one s imulus (one 360◦image) was limi ed o
10second. I he subjec s ound ha hey did no need o use
he whole ime o obse e he s imulus, hey could close hei
eyes o a longe pe iod o ime (0.5s), which was de ec ed
ia eye- acke and led o a educ ion in obse a ion ime.
A e his pe iod passed, he GUI o he e alua ion was
displayed. I consis ed o se e al cubes wi h aded s imulus
in he backg ound. Subjec s had o selec he subjec i ely
pe cei ed image quali y on he i e-le el a ing scale ( he
g een cube ep esen ed he bes -pe cei ed quali y, while he
ed one ep esen ed he lowes image quali y). The ime o he
e alua ion phase was no limi ed. A e he subjec i e a ing
was done, ano he s imulus was shown. This p ocedu e was
epea ed o all es ed images.
On a e age, subjec s ook app oxima ely 21minu es o
comple e he es session. Fou da a iles we e s o ed
om each pa icipan . These iles, in addi ion o pe cei ed
subjec i e sco es, con ain logged head and eye mo emen
da a. The eye- acking da a a e p o ided wi hin he da abase
and could be used in u u e esea ch (e.g., o ain an objec i e
model). To p o ide as much com o as possible o he
subjec s, each o hem could eques a b eak wi hin he es
session. Howe e , no one ook his oppo uni y du ing he
subjec i e expe imen .
C. SUBJECTS
In his s udy, a o al o 36 non-expe subjec s (34 males and
2 emales) om he BUT and 30 (20 males and 10 emales)
om he CTU pa icipa ed in subjec i e OIQA expe imen s.
The age o all subjec s anged om 18 o 29 yea s. Fo he
BUT labo a o y, he mean and median age we e 22.14 and
22 yea s, espec i ely, and o he CTU, he mean was
20.67 and he median 20 yea s. All obse e s had a no mal
o co ec ed- o-no mal ision. Some indi iduals used diop ic
glasses and wo e hem along wi h he headse du ing he
es . In o al, 66 subjec s exceeded he minimum equi emen
11h ps://uni y.com/p oduc s/uni y-engine
FIGURE 3. Co ela ion o he MOS sco es ( om he me ged da ase )
ob ained om subjec i e expe imen s conduc ed a he CTU and BUT
labo a o ies.
FIGURE 4. O e all MOS a ings wi h 95 % CIs ob ained using ACR-HR o
di e en bpp alues. The g een- illed a ea be ween wo ho izon al lines
co esponds o 95 % CI o he hidden e e ence image.
o 28 pe sons as pe ITU-T P.919 ecommenda ion [38].
Fu he mo e, based on his ecommenda ion, no addi ional
sc eening o pa icipan s was necessa y om a s a is ical
s andpoin , gi en he la ge numbe o subjec s in ol ed. As a
esul , no u he pos -sc eening was conduc ed, aside om
he addi ional physiological impac ques ionnai e discussed
in Subsec ion IV-B).
150424 VOLUME 12, 2024
M. Simka e al.: Pe o mance E alua ion o Objec i e Quali y Assessmen Me hods
FIGURE 5. S a is ical pe o mance analysis o objec i e me ics o all da a. AUC, C0and hei signi icance plo s o Di e en /Simila and Be e /Wo se
ROC analyses. The whi e boxes indica e ha he me ic in he ow is signi ican ly be e , black boxes- signi ican ly wo se, and g ay ep esen s ha he
me ic in he ow is nei he signi ican ly be e no wo se compa ed o he me ic in he column. The C0quan i ies he pe cen age success (co ec
ecogni ion) in iden i ying he image wi h he be e quali y om a speci ic pai . The e o ba s a e 95% CI. The speci ic me ics a e ma ked as ollows:
1 - PSNR, 2 - SSIM, 3 - FSIMc, 4 - GMSD, 5 - VIFp, 6 - VMAF, 7 - MS-SSIM, 8 - WS-PSNR, 9 - CPP-PSNR.
IV. RESULTS OF THE STUDY
This sec ion p esen s he ou comes o he subjec i e es s,
encompassing MOS sco es, e alua ion o physiological
impac s on he use , and a compa ison o esul s be ween
labo a o ies. Nex , objec i e and subjec i e quali y assess-
men o 360◦images a e comple ed by ex ensi e s a is ical
analysis.
A. SUBJECTIVE RESULTS
Each pa icipan ’s subjec i e sco e was collec ed and p o-
cessed o de i e he MOS alue, along wi h addi ional
s a is ical da a. The mean sco e o a gi en s imulus was
calcula ed as [39]:
¯uk=1
N
N
X
i=1
uik ,(1)
whe e ukis he subjec i e sco e gi en by obse e i o
s imulus k, and Nis he numbe o obse e s.
The subjec i e es ou pu s om he CTU and BUT
labs we e me ged o join e alua ion, conside ing he
negligible di e ences p edic ed due o HMD o condi-
ions a ia ions [6],[26]. S a is ical analysis, speci ically a
non-pa ame ic Wilcoxon Signed-Rank es [40], alida ed
hese minimal di e ences (p- alue=6.99−16) based on he
MOS sco es o bo h labs. Fig. 3illus a es he o e all
co ela ion be ween all esul ing MOS alues om he
CTU and BUT. Addi ionally, co ela ion coe icien s (wi h
α=0.05), as PLCC=0.9654, Spea man Rank Co ela ion
Coe icien (SRCC) wi h a alue o 0.9534, and also Kendall
Rank Co ela ion Coe icien (KRCC) o 0.8286, we e
calcula ed om bo h labo a o ies, con i ming e y high in e -
lab co ela ion.
The esul s om he subjec i e es s conduc ed in hese
labo a o ies we e me ged and ollowed by a comp ehensi e
analysis o he MOS sco es. To compa e he indi idual com-
p ession algo i hms in e ms o subjec i e quali y assessmen ,
he MOS alues we e e alua ed agains e e ence images.
These p ocessed subjec i e esul s a e shown in Fig. 4, which
isually p esen s he MOS sco es o he i e e e ence
images, accompanied by 95% con idence in e als (CI).
B. PHYSIOLOGICAL IMPACTS
Wi hin he subjec i e expe imen , pa icipan s we e su eyed
using a ques ionnai e o assess he impac o HMD iewing
360◦images on hei physical s a e as an addi ional a ibu e
o he QoE. The pa icipan s comple ed a Simula o Sickness
Ques ionnai e (SSQ) based on [41], cap u ing speci ic phys-
iological pa ame e s (gene al discom o , a igue, headache,
eyes ain, di icul y ocusing, inc eased sali a ion, swea ing,
nausea, di icul y concen a ing, ullness o head, blu ed
ision, dizzy (eyes open), dizzy (eyes closed), e igo,
s omach awa eness, bu ping). Whe eby each symp om was
a ed on a h ee-le el scale (0-none, 1-sligh , 2-mode a e,
3-se e e).
Be o e and a e he subjec i e es , each olun ee
comple ed he ques ionnai e. The esul s showed ha 360◦
images had no signi ican nega i e e ec s on pa icipan s in
compa ison o obse a ions epo ed while wa ching 360◦
ideos [6]. Fo he mos equen ly occu ing impac s, he
e alua ion was conduc ed o he en i e da abase (based
on 66 esponden s). On a e age, a 1-le el impai men in
headache, ullness o head, blu ed ision, and gene al
discom o was no ed by 26% o use s. The mos equen
impac (38%) was a 1-le el shi in eyes ain. None o he
con olled pa ame e s had a 2-le el in luence on mo e han
7% o esponden s.
C. OBJECTIVE ASSESSMENT
Se e al s a is ical analysis me hods we e employed o assess
he pe o mance o he objec i e measu es o each eme ging
image comp ession algo i hm and he e e ence omnidi ec-
ional image ypes. Commonly used s a is ical me hods in he
IQA domain [15],[18],[42] a e PLCC, SRCC, KRCC, and
Roo Mean Squa ed E o (RMSE).
In his s udy, he p ima y analysis in ol ed calcula ing
he A ea unde Cu e (AUC) and co ec classi ica ion
C0 alues using Di e en /Simila and Be e /Wo se s a is ical
ROC me hods [43]. This ROC app oach o e s se e al
enhancemen s o e con en ional s a is ical me hods. I allows
o compa ing objec i e me hods as applied in eal-wo ld
scena ios wi hou he need o map di e en scales, educes
VOLUME 12, 2024 150425
M. Simka e al.: Pe o mance E alua ion o Objec i e Quali y Assessmen Me hods
TABLE 4. Pe o mance o objec i e me hods in e ms o AUC, C0, PLCC, SRCC, and KRCC by e e ence images and by each comp ession algo i hm on he
me ged da ase .
dependency on he e ec s o me ic ange, and enables
he combina ion o o he da ase s, as demons a ed in
his wo k. Addi ionally, i conside s he s a is ical signi -
icance o subjec i e sco es and de e mines he s a is ical
signi icance o pe o mance di e ences. Fu he mo e, he
s a is ical analysis o he s udy was ex ended o include
PLCC (wi hou non-linea mapping), SRCC, and KRCC
me hods. An alpha-le el o α=0.05 was applied in he
s a is ical analysis, and he p- alue was examined o assess
eliabili y. The majo i y o he analysis esul s e en me an
alpha-le el o less han 1 %. Consequen ly, he null hypo he-
sis could be ejec ed in all cases, con i ming hei s a is ical
signi icance.
The Di e en /Simila analysis assesses he me ic’s abili y
o dis inguish signi ican ly di e en pai s om simila ones,
while he Be e /Wo se analysis de e mines i s capabili y
150426 VOLUME 12, 2024
M. Simka e al.: Pe o mance E alua ion o Objec i e Quali y Assessmen Me hods
o co ec ly iden i y he s imulus o highe quali y in a
pai . The C0pe cen age indica es how o en he model
co ec ly ecognizes he highe quali y s imulus. As desc ibed
in [43], Be e /Wo se analysis, along wi h he C0calcula ion,
is conduc ed solely o pai s iden i ied as di e en in he
ini ial s ep o he ROC analysis. The de e mina ion o whe he
pai s a e di e en is based on he cumula i e dis ibu ion
unc ion (CDF) o he no mal dis ibu ion.
G aphical esul s o he ROC analysis, including AUC and
he pe cen age o C0 alues o each objec i e me ic ac oss
he en i e da ase , a e depic ed in Fig. 5. Addi ionally, sum-
ma ized esul s o he pe o mance e alua ion o objec i e
quali y assessmen me hods, epo ed o indi idual e e ence
images and codecs, a e p esen ed in Table 4.
The s a is ical analysis highligh s he SSIM, FSIMc, VIFp,
and MS-SSIM (labeled as No. 2, 3, 5, and 7) as he
op-pe o ming me ics o he es ed use case scena ios.
In he Di e en /Simila me hod (see Fig. 5a), he same
signi icance is no iceable o he SSIM, FSIMc, and MS-
SSIM me ics. On he o he hand, in he Be e /Wo se analysis
(see Fig. 5b), he Signi icance plo emphasizes he SSIM
as he bes pe o me (indica ed by whi e boxes in ow 2),
meaning ha i images a e di e en in a pai , hen his me ic
is he bes a ecognizing he s imulus wi h be e quali y.
No ably, he VIFp s ands ou among o he me ics wi h a
C0 alue (acco ding o Fig. 5c), o e 89%, ep esen ing he
highes pe cen age o decision co ec ness o he s imulus
wi h he be e quali y.
In he con ex o OIQA, he me ics u ilized in his s udy
ailed o su pass he op-pe o ming FR-IQA me ics in
any examined scena io, despi e being speci ically de eloped
o omnidi ec ional con en [18]. Only o images deg aded
by he JPEG codec did he WS-PSNR and CPP-PSNR
exhibi pe o mance compa able o he leading FR-IQA
me hods.
The de ailed esul s o speci ic images and codecs ( e e
o Table 4) indica e ha he pe o mance o me ics a ies
acco ding o image ea u es bu p ima ily depends on he
applied comp ession algo i hm. As expec ed, pe o mances
di e ac oss indi idual e e ence images. The e o e, he
esul s in his con ex a y conside ably among images. Fo
example, he highes alues o he Hokkaido image we e
achie ed by he SSIM, o he Biscayne image by he FSIMc
and GMSD, o he Flowe s image by he FSIMc, VIFp, and
VMAF, o he Telescope image by he PSNR, and o he
T ains image by he SSIM and FSIMc.
The e ec o he applied comp ession algo i hm on he
pe o mance o he analyzed me ics is signi ican . No ably,
he MS-SSIM eme ged as he op pe o me o AVIF and
HEIC comp essed images (highligh ed in bold in Table 4),
and also showed s ong e iciency o JPEG-based codecs,
a leas in one ype o analysis. In con as , he FSIMc
excelled o JPEG and JPEG XL bu was less e ec i e o
AVIF comp ession. Howe e , when conside ing he summa y
esul s o all comp ession algo i hms (see he ‘‘All’’ column
in Table 4), he SSIM and VIFp me ics pe o med bes
o e all, e en hough o he me ics we e mo e e ec i e o
speci ic codecs.
These insigh s in o he pe o mance o objec i e me ics
a e consis en wi h indings om ela ed wo ks. Fo ins ance,
s udies [5] and [16] epo ed ha he VIFp me ic achie ed
he bes o e all esul s acco ding o PLCC and SRCC o
dis o ions caused by JPEG, HEVC, and JPEG2000 codecs.
Simila ly, he FSIMc was iden i ied as he op pe o me
in s udy [19], while he SSIM was he bes in s udy [14],
bo h using he same da abase wi h JPEG, HEVC, and AVC
comp essions. Howe e , he pe o mance o objec i e me ics
o assessing he quali y o images comp essed wi h AVIF and
JPEG XL codecs canno be compa ed o o he esul s, as no
simila s udies ha e been published so a . This gap is one o
he main mo i a ions and con ibu ions o ou wo k.
V. CONCLUSION
Image quali y is a key ac o in achie ing an op imal
expe ience o he use , as low image quali y can deg ade he
imme si e expe ience o 360◦con en and educe he o e all
QoE. The e o e, assessing image quali y is c ucial o he
de elopmen o VR con en and echnologies.
This pape p o ides a unique s udy on objec i e and
subjec i e OIQA o images dis o ed by eme ging and legacy
comp ession algo i hms (AVIF, JPEG XL, HEIC, and JPEG).
The c oss-lab expe imen s, conduc ed by wo independen
labs, in ol ed a subs an ial numbe o 66 subjec s. This
ex ensi e pa icipa ion enhances he s a is ical eliabili y
o he ou comes and enables he e alua ion o po en ial
di e ences be ween he esul s ob ained in he wo labs.
Impo an ly, he s udy con i ms a signi ican co ela ion o
he esul s om bo h labs, al hough di e en HMDs we e
used.
The p ima y ocus was on e alua ing he pe o mance o
con en ional (non-CNN models) objec i e FR me hods when
applied o con en a ec ed by eme ging JPEG XL, HEIC,
and AVIF codecs. This s udy employed a s a is ical ROC
app oach, which, due o i s s eng hs, allowed o a mo e
comp ehensi e analysis han is commonly pe o med. The
esul s con i m he applicabili y o con en ional objec i e
me ics o 360◦images dis o ed by speci ic eme ging
codecs. This was achie ed h ough igo ous s a is ical
analysis, which no only elied on commonly used co ela ion
measu es bu also inco po a ed s a is ically powe ul ROC
me hods (Di e en /Simila and Be e /Wo se) wi h AUC and
C0ou pu s. The analyses e ealed a ying e iciencies o
me ics based on e e ence images. In speci ic applica ion
scena ios, MS-SSIM demons a ed high pe o mance o
AVIF comp ession, while VIFp, MS-SSIM, and FSIMc
excelled o he HEIC codec. Fo JPEG XL, MS-SSIM and
FSIMc we e op imal, and o JPEG, FSIMc, and VIFp p o ed
he mos e ec i e. In summa y, ac oss all scena ios, SSIM,
FSIMc, VIFp, and MS-SSIM consis en ly eme ged as he
bes -pe o ming me ics.
Fu he mo e, his s udy expands upon ou p io esea ch
[28]. I in oduces ex ensi e da a om la ge-scale subjec i e
VOLUME 12, 2024 150427