Performance Evaluation of Objective Quality Assessment Methods for Omnidirectional Images Under Emerging Compressions

Recei ed 21 Augus 2024, accep ed 7 Oc obe 2024, da e o publica ion 11 Oc obe 2024, da e o cu en e sion 24 Oc obe 2024. Digi al Objec Iden i ie 10.1109/ACCESS.2024.3478793 Pe o mance E alua ion o Objec i e Quali y Assessmen Me hods o Omnidi ec ional Images Unde Eme ging Comp essions MAREK SIMKA 1, LADISLAV POLAK 1, (Membe , IEEE), MARTIN NOVOTNY 2, JAN KUFA 1, AND KAREL FLIEGEL 2, (Membe , IEEE) 1Depa men o Radioelec onics, Facul y o Elec ical Enginee ing and Communica ion, B no Uni e si y o Technology, 61600 B no, Czech Republic 2Depa men o Radioelec onics, Facul y o Elec ical Enginee ing, Czech Technical Uni e si y in P ague, 166 27 P ague, Czech Republic Co esponding au ho : Ma ek Simka ([email p o ec ed]) This wo k was suppo ed in pa by he In e nal G an Agency o B no Uni e si y o Technology unde P ojec FEKT-S-23-8191; in pa by he G an Agency o he Czech Technical Uni e si y in P ague unde P ojec SGS23/185/OHK3/3T/13 and P ojec SGS23/186/OHK3/3T/13; and in pa by he Minis y o Educa ion, You h and Spo s o he Czech Republic, unde P ojec LTT20004. ABSTRACT This pape p esen s a unique s udy o objec i e and subjec i e image quali y assessmen (IQA) o 360◦images, dis o ed by eme ging and legacy comp essions - AV1 Image File Fo ma (AVIF), Join Pho og aphic Expe s G oup (JPEG), High E iciency Image File Fo ma (HEIC), JPEG XL om he c oss-lab expe imen s ca ied ou by wo uni e si ies. The pe o mance e alua ion o objec i e con en ional IQA me hods elied on s a is ical da a analysis employing no only adi ional app oaches wi h co ela ion coe icien s bu also ad anced echniques such as Recei e Ope a ing Cha ac e is ic (ROC) analysis, which o e s se e al s a is ical ad an ages. The comp ehensi e s a is ical examina ion p esen ed, u ilizing an ou come om he subjec i e expe imen wi h 66 obse e s, con i med ha he pe o mance o speci ic IQA me ics depends no only on he image ea u es bu also on he applied image comp ession algo i hm. O e all, he s udy examined nine IQA objec i e me ics. Fo ins ance, when e alua ing dis o ion caused by he AVIF and HEIC codecs, he Mul i-Scale S uc u al Simila i y (MS-SSIM) me ic exhibi ed he bes pe o mance, while he Fea u e Simila i y ch ominance (FSIMc) me ic excelled in assessing JPEG-based comp essions. Addi ionally, de ailed ou comes o he subjec i e expe imen s, including eye- acking da a, a e a ailable in a public da ase . This s udy p o ides p e iously unexplo ed insigh s in o he pe o mance o con en ional objec i e me ics o 360◦images deg aded by eme ging image codecs. The p esen ed esul s sugges sui able me ics o speci ic applica ion scena ios (codecs) and, along wi h he da ase , lay he g oundwo k o u he esea ch on new objec i e quali y assessmen me hods o omnidi ec ional images unde eme ging comp essions. INDEX TERMS Omnidi ec ional image (360◦), 360-deg ee con en , i ual eali y, objec i e quali y assessmen , subjec i e es , pe cep ual image quali y. I. INTRODUCTION The p oli e a ion o imme si e echnologies such as i ual and augmen ed eali y (VR and AR) has escala ed he need o supe io omnidi ec ional con en . Omnidi ec ional images, simply called 360◦images, p o ide use s wi h a ull The associa e edi o coo dina ing he e iew o his manusc ip and app o ing i o publica ion was And ea F. Aba e . 360◦pano amic iew o a scene, deli e ing an imme si e and cap i a ing isual expe ience. Recen ly, subs an ial ile sizes o omnidi ec ional images p esen hu dles in s o age, ansmission, and ende ing. To add ess hese challenges e ec i ely, i is c ucial o u ilize e icien comp ession echniques ha p o ide high lexibili y while p ese ing he pe cep ual quali y o he comp essed 360◦images [1]. Mo eo e , o assess he equi ed Quali y o Expe ience (QoE) VOLUME 12, 2024 2024 The Au ho s. This wo k is licensed unde a C ea i e Commons A ibu ion-NonComme cial-NoDe i a i es 4.0 License. Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by-nc-nd/4.0/ 150419 M. Simka e al.: Pe o mance E alua ion o Objec i e Quali y Assessmen Me hods o human obse e s, eliable me hods able o de e mine he isual quali y o omnidi ec ional images a e i al [2],[3]. Nowadays, he assessmen o he 360◦images isual quali y has become a p ominen esea ch a ea wi hin he image quali y assessmen (IQA) domain [4]. Objec i e me ics, which ely on ma hema ical models, a e designed o gauge he pe cep ual quali y o comp essed images. Howe e , mos me ics we e designed and uned o con en ional 2D images, so he e emains a gap o esea ch on he pe o mance o hese me hods o 360◦images [5] o de elopmen o no el me hods uned o his ype o con en s. These me ics ake in o accoun ac o s such as spa ial and angula dependencies, s uc u al simila i y, colo accu acy, ex u e p ese a ion, and isual ideli y. Con e sely, IQA h ough subjec i e means in ol es conduc ing es s in a con olled labo a o y en i onmen , whe e human obse e s p o ide hei subjec i e e alua ions o he pe cei ed quali y o dis o ed images when iewed wi h a head-moun ed display (HMD) [6]. In ecen yea s, subs an ial p og ess has been made in he ield o IQA due o esea ch e o s [7]. One ongoing a ea o esea ch ocuses on de eloping app op ia e me hodologies o assessing he QoE o VR-based con en . The ele ance o his esea ch is e idenced by nume ous pape s add essing IQA in he con ex o 360◦ isual con en , using ei he objec i e [8] o subjec i e [9],[10] me hods. The impo ance o e alua ing he pe cep ual quali y o 360◦images ex ends beyond echnical conside a ions. I has a di ec impac on he o e all use QoE in VR-based applica ions. By com- p ehending he pe cep ual ac o s ha in luence image quali y and le e aging bo h objec i e and subjec i e me hods, i becomes possible o op imize comp ession algo i hms and ansmission echniques. This op imiza ion esul s in he deli e y o high-quali y 360◦images while minimizing he p esence o no iceable deg ada ions. A. RELATED WORKS The impo ance o examining he pe o mance o omnidi- ec ional IQA (OIQA) h ough bo h objec i e and subjec i e me hods is e iden om he conside able amoun o exis ing esea ch. A comp ehensi e o e iew o he ecen s udies de o ed o IQA (including subjec i e es s) o 2D omnidi ec- ional images can be ound in Table 1. A s udy [11] in oduced a es bed o conduc ing subjec i e assessmen s o omnidi ec ional isual con en . This es bed collec ed Mean Opinion Sco e (MOS) a ings, iewing di ec ion da a, and ime spen by up o 48 pa icipan s using he Absolu e Ca ego y Ra ing wi h Hidden Re e ence (ACR-HR) me hod. Hal e alua ed equi ec angula o ma , and hal assessed cubic p ojec ion images. The MOS esul s a o ed he equi ec angula o ma o supe io image quali y. Upenik and e al. in [5] p o ided a s udy compa ing wo ypes o objec i e ull- e e ence (FR) me ics - adi ional o 2D con en and specialized me ics o 360◦images. They comp essed ou s ill images om a ideo da ase by using Join Pho og aphic Expe s G oup (JPEG), JPEG2000. and High E iciency Video Coding (HEVC) codecs. While hey p ima ily used OIQA me ics based on Peak Signal- o- Noise Ra io (PSNR), me ics IQA like S uc u al Simila i y Index Measu e (SSIM), Mul i-Scale S uc u al Simila i y (MS-SSIM), and Visual In o ma ion Fideli y in he pixel domain (VIFp) consis en ly yielded be e quali y a ings. No ably, VIFpou pe o med heo he s, wi ha Pea sonLinea Co ela ion Coe icien (PLCC) alue o 0.8994. In he wo k [12], he esea che s add ess he challenge o quali y assessmen o 360◦images. They c ea ed an OIQA da abase wi h 16 sou ce images and 320 deg aded images, comp essed using JPEG and JPEG2000. They compa ed nine FR-IQA me ics. The s udy highligh s he impo ance o image de ails in he VR expe ience and sugges s using isual saliency o OIQA. The disad an age is only 20 subjec s based on he legacy ITU-R BT.500-11 ecommenda ion designed o TV images, which is no s a is ically su icien o VR con en . Au ho s in [13] ga he ed he la ges 360-deg ee image da ase o da e, known as CVIQD2018, wi h 528 comp essed images using common coding s anda ds. The goal o he esea ch was o es i exis ing OIQA me hods could e alua e comp essed omnidi ec ional image quali y. They used i e no- e e ence (NR) IQA me ics, en adi ional FR-IQA me ics, and h ee PSNR-based me ics o 360◦images. To ensu e usabili y, hey conduc ed subjec i e es ing wi h 20 subjec s using he single-s imulus (SS) me hod. Among he 13 FR-IQA me ics used, SSIM, In o ma ion con en weigh ed SSIM (IW-SSIM), and Visual Saliency-Induced Index (VSI) achie ed he highes consensus wi h subjec i e sco es. The s udy concludes ha ce ain objec i e me ics o 2D images a e s ill alid and obus o 360◦images. In [14], simila o [13], he au ho s used he same image da abase and conside ed p io subjec i e es esul s. Addi ionally, hey explo ed a mul i-channel con olu ional neu al ne wo k (CNN) o no- e e ence quali y assessmen o 360◦images. This model pe o med simila ly o FR me ics and ou pe o med all me ics in he CVIQD2018 da abase o AVC (as H.264) and JPEG comp ession dis o ion. Howe e , he s udy lacked basic in o ma ion abou he es ed sequences (selec ed ypes and amoun s) and elied on sco es om only 20 subjec s acco ding o ITU-R BT.500-11 ecommenda ion. A new pe cep ual quali y me ic o VR-based con en , e med VA-PSNR, was p oposed in [15]. I in eg a es PSNR wi h a saliency map de i ed om eal head mo e- men da a, add essing iewe isual a en ion. This me ic ep esen s an ad ancemen o e he Weigh ed Sphe ical PSNR (WS-PSNR) me ic o 360◦con en . By conside ing iewe in e ac ions and isual a en ion, i acknowledges key ac o s in omnidi ec ional con en pe cep ion. The me ic was es ed on images also dis o ed wi h he HEVC In a ame (HEVC-I) codec as a a ian o comp ess s ill images. Howe e , one limi a ion is he eliance on empi ical isual da a, which may no always be eadily accessible. Despi e 150420 VOLUME 12, 2024 M. Simka e al.: Pe o mance E alua ion o Objec i e Quali y Assessmen Me hods TABLE 1. Su ey o he exis ing s udies on IQA (objec i e & subjec i e) o 2D omnidi ec ional images. his, he s udy o e s a p omising a enue o e alua ing omnidi ec ional con en quali y, wi h u he esea ch needed o alida e i s applicabili y in di e se scena ios and con en ypes. In [16], a no el deep lea ning app oach o OIQA, called DeepVR-IQA, was in oduced. The model demons a es supe io pe o mance, accu a ely es ima ing pixel quali y and deli e ing an excellen o e all assessmen . Achie ing his high pe o mance necessi a ed aining on subjec i e a ings o up o 720 s imulus, esul ing in a s ong co ela ion wi h subjec i e sco es, e ec i ely cap u ing image cha ac e is ics a ec ing isual quali y. A d awback o his me hod is i s demand o a subs an ial olume o aining da a, which could be challenging in ce ain applica ions. The pape [17] examines he impac o display cha ac- e is ics and human a en ion on 360◦images, c ea ing a da ase ha includes subjec i e e alua ions o hese images. The au ho s p opose he so-called CPBQA amewo k, which combines he Blind/Re e enceless Image Spa ial Quali y E alua o (BRISQUE) and he In eg a ed Local Na u al Image Quali y E alua o (IL-NIQE). This amewo k in eg a es global and local quali y ea u es o cubemaps o blind quali y assessmen . Fu u e esea ch in his a ea will ocus on de eloping mo e e icien ea u e ex ac ion me hods and explo ing he e ec o ho spo maps on image quali y. The pu pose o wo k [18] was o unde s and human pe - cep ion o 360◦image quali y. Subjec i e a ings and iewing beha io da a unde di e se condi ions we e collec ed, ocusing on he co ela ion be ween iewing condi ions, non- uni o m dis o ions, and subjec i e sco es. They in oduced a no el OIQA model based on a ResNe -50 CNN, inco po- a ing mul i-scale ea u e ex ac ion and quali y p edic ion modules. In compa ison wi h o he me ics, hei p oposed model ou pe o med in mos cases, excep o scena ios in ol ing Gaussian noise deg ada ion. Au ho s in [19] and [20] es ablished a new model o blind/NR OIQA (BOIQA). A colo desc ip o ha e lec s colo in o ma ion was used. O e all, he model should exhibi a g adien e lec ion as global s uc u al ea u es and g ay le els wi h local s uc u al ea u es. They did no ga he hei own subjec i e a ings bu es ed he model on [12],[13] da a. The BOIQA wi h suppo ec o eg ession is p omising, bu he e is scope o mo e alida ion wi h a mo e obus model [22]. Duan and e al. in he s udy [21] ocused on OIQA o speci ic s i ching- ela ed in luences. The aw 12 images we e co up ed du ing he s i ching p ocess namely colo , geome - ic, blu , and ghos ing dis o ion. Following hese dis o ions in e wined wi h he acquisi ion o omnidi ec ional con en and subjec i e a ings by 20 esponden s, hey p oposed a model sepa a ely o FR and NR OIQA. The impac o comp ession was no conside ed. B. ORIGINAL CONTRIBUTIONS F om he abo e elabo a ed s a e-o - he-a (SoA), i is possible o de i e he ollowing conclusions. Recen ly, he OIQA-based s udies s ill do no conside eme ging image comp ession algo i hms and ela ed speci ic comp ession a i ac s, namely JPEG XL [23], High E iciency Image File Fo ma (HEIC) [24] and AV1 Image File Fo ma (AVIF) [25]. Nex , an analysis o con en ional (non-CNN models) objec i e me ics o OIQA wi h ex ensi e subjec i e es s has no been p o ided so a . In pa icula , ho ough s a is ical pe o mance e alua ion beyond co ela ion coe icien s is also lacking. Conce ning all he a o emen ioned gaps in he p io a , he main con ibu ions o ou wo k a e summa ized as ollows: •Applicabili y alida ion o eme ging image comp ession algo i hms (HEIC, JPEG XL, AVIF) o 360◦images. •E alua ion o objec i e con en ional IQA me hods in e ms o 360◦images co up ed by eme ging codecs. VOLUME 12, 2024 150421 M. Simka e al.: Pe o mance E alua ion o Objec i e Quali y Assessmen Me hods •Ex ensi e subjec i e expe imen s conduc ed by wo lab- o a o ies, coupled wi h s a is ical analysis by Recei e Ope a ing Cha ac e is ic (ROC) me hodology, ep esen an ad anced app oach o pe o mance e alua ion. •In luence e alua ion o he HMDs and es condi ions on he QoE by c oss-lab s udy (in e -lab co ela ion [26]). •P o iding new anno a ed da ase ,1including MOS alues, aw sco es, and eye- acking da a, as a igo ous ex ension o ou p e ious wo k [27]. To he bes o ou knowledge, a simila s udy wi h he de ined objec i es has no been p o ided so a . C. ORGANIZATION OF THE PAPER The s uc u e o he pape is ou lined as ollows. In oduc ion o he s udy p o ided in his pape , he elabo a ion o he SoA, and he main con ibu ions o his wo k a e included in Sec ion I. The cha ac e is ics o e e ence 360◦images, he codecs con igu a ion, and objec i e quali y assessmen a e p esen ed in Sec ion II. The comple e subjec i e es me hodology is desc ibed in Sec ion III. The main esul s o bo h objec i e and subjec i e s udies a e gi en in Sec ion IV. Finally, Sec ion Vconcludes he pape . II. OBJECTIVE QUALITY ASSESSMENT OF 360◦IMAGES This sec ion p o ides de ails on he e e ence images, he image codecs, and hei con igu a ions and speci ica ions o he objec i e quali y me ics applied. A. OMNIDIRECTIONAL IMAGE DATA In his s udy, i e ca e ully selec ed e e ence 360◦images and hei dis o ed a ian s we e used om he open da abase OMNIQAD [28]. A p e iew o hese images is depic ed in Fig. 1. The Sigh s image was used only in he aining session o he subjec i e es . The de ailed p ope ies o he e e ence images, including esolu ion and spa ial in o ma ion (SI) [29], a e a ailable in Table 2. All hese 360◦images we e selec ed ollowing expe iewing sessions wi h di e en ea u es, esul ing in di e se esponses om he used image codecs. Acco ding o he a ge bi pe pixel (bpp), see Table 3, a o al o 100 images was e alua ed, as he e we e 19 dis o ed a ian s o each e e ence image, and he e e ence image was also included. B. CODING CONFIGURATIONS As men ioned abo e, comp ession was pe o med wi h one legacy s ill image codec (JPEG) and h ee eme ging comp ession algo i hms, namely: JPEG XL, HEIC, and AVIF. The JPEG XL is he mos ecen successo o JPEG wi h highe lexibili y and comp ession e iciency. HEIC and AVIF, based on licensed HEVC and license- ee AV1 ideo codecs espec i ely, a e eme ging image o ma s. I was shown ha o con en ional 2D images, JPEG XL, HEIC, and 1h ps://zenodo.o g/doi/10.5281/zenodo.7607070 2h ps://gi hub.com/Telecommunica ion-Telemedia-Assessmen /SITI FIGURE 1. Snapsho s o he e e ence 360◦images (*Image used only in he aining session o subjec i e p ocedu e). TABLE 2. Speci ica ions o he e e ence omnidi ec ional images. TABLE 3. Desc ip ion o he selec ed a ian s o comp essed images. AVIF excel in achie ing small ile sizes while p ese ing he high isual quali y o he images [23],[24],[25]. As a ep esen a i e o abled JPEG-based comp ession, he JPEG XT [30] e e ence so wa e3has been used. Image comp ession wi h he AVIF and JPEG XL codecs has been ealized using he liba codec and libjxl lib a ies in FFmpeg.4 In he case o he HEIC codec, he GIMP52.10.32 g aphics edi o was used. I mus be no ed ha he selec ed imple- men a ions o hese image comp ession s anda ds ha e been used o ob ain ep esen a i e image deg ada ion, and no o e alua e he ac ual codec pe o mance. The con igu a ions and se ings o he encode s we e mainly ela ed o de ining he inpu image, he coding lib a y, and he quan iza ion pa ame e . Basically, i was a de aul se ing o each implemen a ion, hus o he pa ame e s emained unchanged. 3h ps://jpeg.o g/jpegx /so wa e.h ml 4h ps://www. mpeg.o g/ 5h ps://www.gimp.o g/ 150422 VOLUME 12, 2024 M. Simka e al.: Pe o mance E alua ion o Objec i e Quali y Assessmen Me hods C. OBJECTIVE QUALITY METRICS In his s udy, we u ilized a ious objec i e quali y me ics o quan i y he dis o ion in he examined 360◦images. Key among hese we e he e e ence me ics, speci ically FR- IQA me ics, including PSNR, SSIM, MS-SSIM, Fea u e Simila i y ch ominance (FSIMc), G adien Magni ude Sim- ila i y De ia ion (GMSD), VIFp, and Video Mul i-Me hod Assessmen Fusion (VMAF). The PSNR, SSIM [31], and VMAF6sco es we e ob ained by FFmpeg amewo k using he la i and lib ma lib a ies, espec i ely. The MS-SSIM [32] and VIFp [33] me ics we e compu ed in MATLAB using an a ailable sc ip .7The FSIMc me ic was used acco ding o [34], and GMSD was u ilized in i s o iginal o m as in oduced in [35]. The WS-PSNR [36] and C as e Pa abolic P ojec ion PSNR (CPP-PSNR) [37] we e chosen as special FR-OIQA me ics ailo ed o equi ec angula p ojec ion con en . These objec i e me hods ha e been applied by he command ool.8 The selec ion o he applied objec i e IQA me hods is based on he mos commonly used con en ional (non-CNN models) me ics men ioned in he ele an p io a [5], [11],[12],[13],[14],[15],[16],[17],[18],[19],[20], [21] dealing wi h 360◦image quali y (see in Subsec ion I-A). This will es ablish he applicabili y and pe o mance o hese me ics e en o images comp essed wi h eme ging codecs. III. SUBJECTIVE QUALITY ASSESSMENT OF 360◦IMAGES This sec ion desc ibes he subjec i e es p ocedu e, including de ails o he es condi ions, equipmen , and subjec s. A. EXPERIMENTAL ENVIRONMENT AND SETUP The subjec i e es s we e conduc ed in wo labo a o ies wi h con olled iewing condi ions. One is he Depa men o Radioelec onics (DREL) a he B no Uni e si y o Technology (BUT), while he o he is DREL a he Czech Technical Uni e si y in P ague (CTU). Among o he s, such a c oss-lab s udy acili a es a mo e complex e alua ion o a ious ac o s, such as HMD used, on he o e all use s’ QoE. In he subjec i e expe imen s ealized a he BUT, he Oculus Ri S9HMD was u ilized as a VR headse . I ea u es an LCD panel wi h a esolu ion o 1280×1440 (pe eye), a e esh a e o 80Hz, and a Field o View (FoV) o 88◦. The headse was connec ed o a lap op unning Oculus so wa e. To ensu e com o able iewing o images om a ious di ec ions, obse e s we e p o ided wi h swi el chai s. A he CTU he HTC Vi e P o Eye10 HMD was used. This HMD has an OLED display wi h a esolu ion o 1440×1600 (pe eye), a e esh a e o 90Hz, and a isible FoV o 110◦. A headse is also equipped wi h eye- acking senso s. The 6h ps://gi hub.com/Ne lix/ ma 7h ps://gi hub.com/sa a ab/image-quali y- ools 8h ps://gi hub.com/Samsung/360 ools 9h ps:// -compa e.com/headse /oculus i s 10h ps://www. i e.com/au/p oduc / i e-p o-eye/specs/ used HMD was connec ed o a PC (HP Z230 wo ks a ion wi h RTX 2060 GPU) unning a es applica ion. B. SUBJECTIVE EXPERIMENT PROCEDURE Following heITU-T P.919[38] ecommenda ion,we adop ed he ACR-HR me hodology wi h a i e-le el a ing scale (1-bad, 2-poo , 3- ai , 4-good, and 5-excellen ) o IQA o 360◦images. In con as o con en ional subjec i e expe - imen s (wi h PC moni o , TV display, e c.), en i onmen al ac o s such as iewing dis ance and ligh ing condi ions a e no equi ed o be conside ed. The expe imen p ocedu e, in bo h labs, consis s o wo pa s - aining and es sessions. 1) BUT The aining session aimed o in oduce he subjec i e expe imen and amilia ize he pa icipan s wi h he VR echnology and omnidi ec ional images. The es p inciple and he pu pose o he expe imen we e explained ini ially. Subsequen ly, a ques ionnai e ga he ed basic in o ma ion abou pa icipan s (age, gende , wea ing glasses, e c.). Nex , a ision sc eening assessed isual acui y using he Snellen cha and colo sensi i i y h ough he Ishiha a es . A e amilia izing wi h he HMD, each pa icipan iewed i e sample images showcasing 360◦con en , speci ically comp essed a ian s o he Sigh s image (see Fig. 1* )), se ing as aining examples. This p ocess helped pa icipan s unde s and how o p o ide pe cep ual quali y a ings. The aining session is ollowed by he es one. I s imeline is depic ed in Fig. 2. Each es s imulus ( es ed image), including he e e ence, was p esen ed one a a ime in a pseudo- andom o de . Obse e s had a maximum iewing ime o 15seconds o less, depending on he obse e ’s decision. Subsequen ly, a o ing pe iod was ini i- a ed, displaying a a ing scale on a g ay backg ound. Du ing his ime, subjec s e bally communica ed hei subjec i e sco es, which we e eco ded by he expe imen e on a PC. Sequence ansi ions we e au oma ed a e 15 seconds o manually ad anced by he expe imen e using a con olle , ensu ing consis en condi ions o each subjec . On a e age, pa icipan s ook 19minu es o comple e he subjec i e expe imen es session. Obse e s had he op ion o expedi e he p ocess by submi ing hei quali y sco e be o e he ime limi . Mo eo e , subjec s could eques a pause a any ime o a b eak and es , e ec i ely a oiding iewing a igue. FIGURE 2. The imeline o he subjec i e es wi h he ACR-HR me hod (TS1 ep esen s he i s es ed s imulus, TS2 he second es ed s imulus). Du ing he o ing in e al, a g ay backg ound wi h a a ing scale was displayed. VOLUME 12, 2024 150423 M. Simka e al.: Pe o mance E alua ion o Objec i e Quali y Assessmen Me hods 2) CTU As a he BUT, he es was di ided in o wo pa s. Du ing he aining session, he subjec s we e sea ed (a s a iona y chai was used a he CTU), in oduced o he es p inciple, and amilia ized wi h he con ols o he subjec i e expe imen applica ion. Subsequen ly, he subjec s illed ou he ques ionnai e ga he ing hei pe sonal in o ma ion (iden ical o he BUT). In opposi ion o he BUT, no ision sc eenings we e held wi h he subjec s du ing he aining session a he CTU, and he subjec s we e asked o epo e en ual ision de iciencies i known. Subjec s wi h diagnosed ision de iciencies we e no included in he s udy. The en i e es session was au oma ed wi h an applica ion de eloped a he CTU using Uni y Engine.11 Each pa icipan was shown comp essed 360◦con en (also wi h hidden e e ence) in andomized o de and one a a ime. The display ime o one s imulus (one 360◦image) was limi ed o 10second. I he subjec s ound ha hey did no need o use he whole ime o obse e he s imulus, hey could close hei eyes o a longe pe iod o ime (0.5s), which was de ec ed ia eye- acke and led o a educ ion in obse a ion ime. A e his pe iod passed, he GUI o he e alua ion was displayed. I consis ed o se e al cubes wi h aded s imulus in he backg ound. Subjec s had o selec he subjec i ely pe cei ed image quali y on he i e-le el a ing scale ( he g een cube ep esen ed he bes -pe cei ed quali y, while he ed one ep esen ed he lowes image quali y). The ime o he e alua ion phase was no limi ed. A e he subjec i e a ing was done, ano he s imulus was shown. This p ocedu e was epea ed o all es ed images. On a e age, subjec s ook app oxima ely 21minu es o comple e he es session. Fou da a iles we e s o ed om each pa icipan . These iles, in addi ion o pe cei ed subjec i e sco es, con ain logged head and eye mo emen da a. The eye- acking da a a e p o ided wi hin he da abase and could be used in u u e esea ch (e.g., o ain an objec i e model). To p o ide as much com o as possible o he subjec s, each o hem could eques a b eak wi hin he es session. Howe e , no one ook his oppo uni y du ing he subjec i e expe imen . C. SUBJECTS In his s udy, a o al o 36 non-expe subjec s (34 males and 2 emales) om he BUT and 30 (20 males and 10 emales) om he CTU pa icipa ed in subjec i e OIQA expe imen s. The age o all subjec s anged om 18 o 29 yea s. Fo he BUT labo a o y, he mean and median age we e 22.14 and 22 yea s, espec i ely, and o he CTU, he mean was 20.67 and he median 20 yea s. All obse e s had a no mal o co ec ed- o-no mal ision. Some indi iduals used diop ic glasses and wo e hem along wi h he headse du ing he es . In o al, 66 subjec s exceeded he minimum equi emen 11h ps://uni y.com/p oduc s/uni y-engine FIGURE 3. Co ela ion o he MOS sco es ( om he me ged da ase ) ob ained om subjec i e expe imen s conduc ed a he CTU and BUT labo a o ies. FIGURE 4. O e all MOS a ings wi h 95 % CIs ob ained using ACR-HR o di e en bpp alues. The g een- illed a ea be ween wo ho izon al lines co esponds o 95 % CI o he hidden e e ence image. o 28 pe sons as pe ITU-T P.919 ecommenda ion [38]. Fu he mo e, based on his ecommenda ion, no addi ional sc eening o pa icipan s was necessa y om a s a is ical s andpoin , gi en he la ge numbe o subjec s in ol ed. As a esul , no u he pos -sc eening was conduc ed, aside om he addi ional physiological impac ques ionnai e discussed in Subsec ion IV-B). 150424 VOLUME 12, 2024 M. Simka e al.: Pe o mance E alua ion o Objec i e Quali y Assessmen Me hods FIGURE 5. S a is ical pe o mance analysis o objec i e me ics o all da a. AUC, C0and hei signi icance plo s o Di e en /Simila and Be e /Wo se ROC analyses. The whi e boxes indica e ha he me ic in he ow is signi ican ly be e , black boxes- signi ican ly wo se, and g ay ep esen s ha he me ic in he ow is nei he signi ican ly be e no wo se compa ed o he me ic in he column. The C0quan i ies he pe cen age success (co ec ecogni ion) in iden i ying he image wi h he be e quali y om a speci ic pai . The e o ba s a e 95% CI. The speci ic me ics a e ma ked as ollows: 1 - PSNR, 2 - SSIM, 3 - FSIMc, 4 - GMSD, 5 - VIFp, 6 - VMAF, 7 - MS-SSIM, 8 - WS-PSNR, 9 - CPP-PSNR. IV. RESULTS OF THE STUDY This sec ion p esen s he ou comes o he subjec i e es s, encompassing MOS sco es, e alua ion o physiological impac s on he use , and a compa ison o esul s be ween labo a o ies. Nex , objec i e and subjec i e quali y assess- men o 360◦images a e comple ed by ex ensi e s a is ical analysis. A. SUBJECTIVE RESULTS Each pa icipan ’s subjec i e sco e was collec ed and p o- cessed o de i e he MOS alue, along wi h addi ional s a is ical da a. The mean sco e o a gi en s imulus was calcula ed as [39]: ¯uk=1 N N X i=1 uik ,(1) whe e ukis he subjec i e sco e gi en by obse e i o s imulus k, and Nis he numbe o obse e s. The subjec i e es ou pu s om he CTU and BUT labs we e me ged o join e alua ion, conside ing he negligible di e ences p edic ed due o HMD o condi- ions a ia ions [6],[26]. S a is ical analysis, speci ically a non-pa ame ic Wilcoxon Signed-Rank es [40], alida ed hese minimal di e ences (p- alue=6.99−16) based on he MOS sco es o bo h labs. Fig. 3illus a es he o e all co ela ion be ween all esul ing MOS alues om he CTU and BUT. Addi ionally, co ela ion coe icien s (wi h α=0.05), as PLCC=0.9654, Spea man Rank Co ela ion Coe icien (SRCC) wi h a alue o 0.9534, and also Kendall Rank Co ela ion Coe icien (KRCC) o 0.8286, we e calcula ed om bo h labo a o ies, con i ming e y high in e - lab co ela ion. The esul s om he subjec i e es s conduc ed in hese labo a o ies we e me ged and ollowed by a comp ehensi e analysis o he MOS sco es. To compa e he indi idual com- p ession algo i hms in e ms o subjec i e quali y assessmen , he MOS alues we e e alua ed agains e e ence images. These p ocessed subjec i e esul s a e shown in Fig. 4, which isually p esen s he MOS sco es o he i e e e ence images, accompanied by 95% con idence in e als (CI). B. PHYSIOLOGICAL IMPACTS Wi hin he subjec i e expe imen , pa icipan s we e su eyed using a ques ionnai e o assess he impac o HMD iewing 360◦images on hei physical s a e as an addi ional a ibu e o he QoE. The pa icipan s comple ed a Simula o Sickness Ques ionnai e (SSQ) based on [41], cap u ing speci ic phys- iological pa ame e s (gene al discom o , a igue, headache, eyes ain, di icul y ocusing, inc eased sali a ion, swea ing, nausea, di icul y concen a ing, ullness o head, blu ed ision, dizzy (eyes open), dizzy (eyes closed), e igo, s omach awa eness, bu ping). Whe eby each symp om was a ed on a h ee-le el scale (0-none, 1-sligh , 2-mode a e, 3-se e e). Be o e and a e he subjec i e es , each olun ee comple ed he ques ionnai e. The esul s showed ha 360◦ images had no signi ican nega i e e ec s on pa icipan s in compa ison o obse a ions epo ed while wa ching 360◦ ideos [6]. Fo he mos equen ly occu ing impac s, he e alua ion was conduc ed o he en i e da abase (based on 66 esponden s). On a e age, a 1-le el impai men in headache, ullness o head, blu ed ision, and gene al discom o was no ed by 26% o use s. The mos equen impac (38%) was a 1-le el shi in eyes ain. None o he con olled pa ame e s had a 2-le el in luence on mo e han 7% o esponden s. C. OBJECTIVE ASSESSMENT Se e al s a is ical analysis me hods we e employed o assess he pe o mance o he objec i e measu es o each eme ging image comp ession algo i hm and he e e ence omnidi ec- ional image ypes. Commonly used s a is ical me hods in he IQA domain [15],[18],[42] a e PLCC, SRCC, KRCC, and Roo Mean Squa ed E o (RMSE). In his s udy, he p ima y analysis in ol ed calcula ing he A ea unde Cu e (AUC) and co ec classi ica ion C0 alues using Di e en /Simila and Be e /Wo se s a is ical ROC me hods [43]. This ROC app oach o e s se e al enhancemen s o e con en ional s a is ical me hods. I allows o compa ing objec i e me hods as applied in eal-wo ld scena ios wi hou he need o map di e en scales, educes VOLUME 12, 2024 150425 M. Simka e al.: Pe o mance E alua ion o Objec i e Quali y Assessmen Me hods TABLE 4. Pe o mance o objec i e me hods in e ms o AUC, C0, PLCC, SRCC, and KRCC by e e ence images and by each comp ession algo i hm on he me ged da ase . dependency on he e ec s o me ic ange, and enables he combina ion o o he da ase s, as demons a ed in his wo k. Addi ionally, i conside s he s a is ical signi - icance o subjec i e sco es and de e mines he s a is ical signi icance o pe o mance di e ences. Fu he mo e, he s a is ical analysis o he s udy was ex ended o include PLCC (wi hou non-linea mapping), SRCC, and KRCC me hods. An alpha-le el o α=0.05 was applied in he s a is ical analysis, and he p- alue was examined o assess eliabili y. The majo i y o he analysis esul s e en me an alpha-le el o less han 1 %. Consequen ly, he null hypo he- sis could be ejec ed in all cases, con i ming hei s a is ical signi icance. The Di e en /Simila analysis assesses he me ic’s abili y o dis inguish signi ican ly di e en pai s om simila ones, while he Be e /Wo se analysis de e mines i s capabili y 150426 VOLUME 12, 2024 M. Simka e al.: Pe o mance E alua ion o Objec i e Quali y Assessmen Me hods o co ec ly iden i y he s imulus o highe quali y in a pai . The C0pe cen age indica es how o en he model co ec ly ecognizes he highe quali y s imulus. As desc ibed in [43], Be e /Wo se analysis, along wi h he C0calcula ion, is conduc ed solely o pai s iden i ied as di e en in he ini ial s ep o he ROC analysis. The de e mina ion o whe he pai s a e di e en is based on he cumula i e dis ibu ion unc ion (CDF) o he no mal dis ibu ion. G aphical esul s o he ROC analysis, including AUC and he pe cen age o C0 alues o each objec i e me ic ac oss he en i e da ase , a e depic ed in Fig. 5. Addi ionally, sum- ma ized esul s o he pe o mance e alua ion o objec i e quali y assessmen me hods, epo ed o indi idual e e ence images and codecs, a e p esen ed in Table 4. The s a is ical analysis highligh s he SSIM, FSIMc, VIFp, and MS-SSIM (labeled as No. 2, 3, 5, and 7) as he op-pe o ming me ics o he es ed use case scena ios. In he Di e en /Simila me hod (see Fig. 5a), he same signi icance is no iceable o he SSIM, FSIMc, and MS- SSIM me ics. On he o he hand, in he Be e /Wo se analysis (see Fig. 5b), he Signi icance plo emphasizes he SSIM as he bes pe o me (indica ed by whi e boxes in ow 2), meaning ha i images a e di e en in a pai , hen his me ic is he bes a ecognizing he s imulus wi h be e quali y. No ably, he VIFp s ands ou among o he me ics wi h a C0 alue (acco ding o Fig. 5c), o e 89%, ep esen ing he highes pe cen age o decision co ec ness o he s imulus wi h he be e quali y. In he con ex o OIQA, he me ics u ilized in his s udy ailed o su pass he op-pe o ming FR-IQA me ics in any examined scena io, despi e being speci ically de eloped o omnidi ec ional con en [18]. Only o images deg aded by he JPEG codec did he WS-PSNR and CPP-PSNR exhibi pe o mance compa able o he leading FR-IQA me hods. The de ailed esul s o speci ic images and codecs ( e e o Table 4) indica e ha he pe o mance o me ics a ies acco ding o image ea u es bu p ima ily depends on he applied comp ession algo i hm. As expec ed, pe o mances di e ac oss indi idual e e ence images. The e o e, he esul s in his con ex a y conside ably among images. Fo example, he highes alues o he Hokkaido image we e achie ed by he SSIM, o he Biscayne image by he FSIMc and GMSD, o he Flowe s image by he FSIMc, VIFp, and VMAF, o he Telescope image by he PSNR, and o he T ains image by he SSIM and FSIMc. The e ec o he applied comp ession algo i hm on he pe o mance o he analyzed me ics is signi ican . No ably, he MS-SSIM eme ged as he op pe o me o AVIF and HEIC comp essed images (highligh ed in bold in Table 4), and also showed s ong e iciency o JPEG-based codecs, a leas in one ype o analysis. In con as , he FSIMc excelled o JPEG and JPEG XL bu was less e ec i e o AVIF comp ession. Howe e , when conside ing he summa y esul s o all comp ession algo i hms (see he ‘‘All’’ column in Table 4), he SSIM and VIFp me ics pe o med bes o e all, e en hough o he me ics we e mo e e ec i e o speci ic codecs. These insigh s in o he pe o mance o objec i e me ics a e consis en wi h indings om ela ed wo ks. Fo ins ance, s udies [5] and [16] epo ed ha he VIFp me ic achie ed he bes o e all esul s acco ding o PLCC and SRCC o dis o ions caused by JPEG, HEVC, and JPEG2000 codecs. Simila ly, he FSIMc was iden i ied as he op pe o me in s udy [19], while he SSIM was he bes in s udy [14], bo h using he same da abase wi h JPEG, HEVC, and AVC comp essions. Howe e , he pe o mance o objec i e me ics o assessing he quali y o images comp essed wi h AVIF and JPEG XL codecs canno be compa ed o o he esul s, as no simila s udies ha e been published so a . This gap is one o he main mo i a ions and con ibu ions o ou wo k. V. CONCLUSION Image quali y is a key ac o in achie ing an op imal expe ience o he use , as low image quali y can deg ade he imme si e expe ience o 360◦con en and educe he o e all QoE. The e o e, assessing image quali y is c ucial o he de elopmen o VR con en and echnologies. This pape p o ides a unique s udy on objec i e and subjec i e OIQA o images dis o ed by eme ging and legacy comp ession algo i hms (AVIF, JPEG XL, HEIC, and JPEG). The c oss-lab expe imen s, conduc ed by wo independen labs, in ol ed a subs an ial numbe o 66 subjec s. This ex ensi e pa icipa ion enhances he s a is ical eliabili y o he ou comes and enables he e alua ion o po en ial di e ences be ween he esul s ob ained in he wo labs. Impo an ly, he s udy con i ms a signi ican co ela ion o he esul s om bo h labs, al hough di e en HMDs we e used. The p ima y ocus was on e alua ing he pe o mance o con en ional (non-CNN models) objec i e FR me hods when applied o con en a ec ed by eme ging JPEG XL, HEIC, and AVIF codecs. This s udy employed a s a is ical ROC app oach, which, due o i s s eng hs, allowed o a mo e comp ehensi e analysis han is commonly pe o med. The esul s con i m he applicabili y o con en ional objec i e me ics o 360◦images dis o ed by speci ic eme ging codecs. This was achie ed h ough igo ous s a is ical analysis, which no only elied on commonly used co ela ion measu es bu also inco po a ed s a is ically powe ul ROC me hods (Di e en /Simila and Be e /Wo se) wi h AUC and C0ou pu s. The analyses e ealed a ying e iciencies o me ics based on e e ence images. In speci ic applica ion scena ios, MS-SSIM demons a ed high pe o mance o AVIF comp ession, while VIFp, MS-SSIM, and FSIMc excelled o he HEIC codec. Fo JPEG XL, MS-SSIM and FSIMc we e op imal, and o JPEG, FSIMc, and VIFp p o ed he mos e ec i e. In summa y, ac oss all scena ios, SSIM, FSIMc, VIFp, and MS-SSIM consis en ly eme ged as he bes -pe o ming me ics. Fu he mo e, his s udy expands upon ou p io esea ch [28]. I in oduces ex ensi e da a om la ge-scale subjec i e VOLUME 12, 2024 150427