Perceptually Lossless Compression with Error Concealment for Periscope and Sonar Videos

Signal & Image Processing: An International Journal (SIPIJ) Vol.10, No.2, April 2019 DOI: 10.5121/sipij.2019.10201 1 PERCEPTUALLY LOSSLESS COMPRESSION WITH ERROR CONCEALMENT FOR PERISCOPE AND SONAR VIDEOS Chiman Kwan 1 , Jude Larkin 1 , Bence Budavari 1 , Eric Shang 1 , and Trac D. Tran 2 1 Applied Research LLC, Rockville, Maryland, USA 2 The Johns Hopkins University, Baltimore, Maryland, USA A BSTRACT We present a video compression framework that has two key features. First, we aim at achieving perceptually lossless compression for low frame rate videos (6 fps). Four well-known video codecs in the literature have been evaluated and the performance was assessed using four well-known performance metrics. Second, we investigated the impact of error concealment algorithms for handling corrupted pixels due to transmission errors in communication channels. Extensive experiments using actual videos have been performed to demonstrate the proposed framework. K EYWORDS Perceptually lossless compression; error recovery; maritime and sonar videos 1. I NTRODUCTION Video compression has been widely used in many commercial and military applications [1][2] . Well-known video codecs include X264 [3] and X265 [4].In some applications such as security monitoring videos where video quality is of prime importance, people are still using lossless image compression algorithms such as JPEG [5] and JPEG-2000 [6] for taking snapshots for storage. Some recent algorithms such as X264 [3] and X265 [4] also provide lossless compression options. This may be too conservative, as lossless compression can only achieve two to four times of compression. JPEG, X264, and X265 are discrete cosine transform (DCT) based algorithms and JPEG-2000 is wavelet based [7]. About 15 years ago, there were some development in DCT based algorithms where overlapped blocks known as lapped transforms (LT) were used to further improve the compression [8]. In the past few years, researchers at Xiph have incorporated LT [8] into an open source video codec known as Daala [9]. Transmission of compressed videos over a wireless channel is susceptible to natural or manmade interferences. A lost bit may damage a whole macro block. Although there are mechanisms such as flexible macro block ordering (FMO) in video codecs that can deal with lost packets to certain extent, it is still not enough, as errors tend to propagate to future frames [10]. Another common way is to insert extra bits to protect the bit stream. However, this will limit the amount of information that can be transmitted over bandwidth constrained channels. Signal & Image Processing: An International Journal (SIPIJ) Vol.10, No.2, April 2019 2 In our earlier study [11], we proposed an error concealment scheme for still image compression. It was observed that lost information can be recovered via error concealment algorithms. No additional bandwidth will be used to protect the bit streams. The error concealment is done at the receiving end, which usually has more computational power. Originally, our sponsor required us to achieve perceptually lossless compression with 8:1 compression ratio for videos at frame rate of 6 frames per second. In our companion paper [11], we have demonstrated that 10 to 1 perceptually lossless compression can be achieved for still images. Hence, we aim at 20 to 1 compression for videos in this research, as videos have more redundancy than still images. We propose to apply four state-of-the-art video codecs (J2K, X264, X265 and Daala) to compress videos. Four performance metrics were used in our study. Moreover, since some applications use wireless channels to transmit compressed videos back to a control center, there are corrupted pixels due to channel errors. We also investigate the possibility of using error concealment techniques to repair the corrupted pixels. A shortened version of this paper was presented in a conference [12]. We have significantly expand our paper. First, we included more compression studies. One periscope and one sonar videos were used in our studies, as they are both of interest to our sponsor. In each study, we have included new tables summarizing the performance metrics of the compression algorithms at 20 to 1 compression or equivalently at a compression ratio of 0.05. Second, we have included more error concealment studies. Two algorithms were compared and objective and visual comparisons were included. Although the compression codecs are not new, the key contribution of our project is to integrate three components (compression, error concealment, and perceptually lossless evaluation) in the video compression system into a single framework. Our paper is organized as follows. Section 2 summarizes the technical approach and its components. Section 3 summarizes all the experiments using actual images that are of interest to our customer. Finally, concluding remarks will be given in Section 4. 2. T ECHNICAL A PPROACH 2.1. Proposed Video Compression and Error Recovery Framework In this research, we will perform objective evaluations of different compression algorithms in the literature. This is to ensure that we will deliver the best approach to our customer. Our overall technical approach can be summarized as follows. First, we will briefly review the literature to determine the compression algorithms available in the market. At the same time, we will describe four performance metrics for algorithm evaluation. The focus will be on metrics that can better model human perception. We will also mention the error resilient algorithms and error concealment techniques. Second, we will obtain realistic periscope and sonar images for algorithm evaluation. Third, we will apply the various compression algorithms to the collected images and generate various performance metrics. Finally, we will also apply advanced algorithms to deal with the corrupted pixels due to channel errors. Signal & Image Processing: An International Journal (SIPIJ) Vol.10, No.2, April 2019 3 Our proposed approach is shown in the following diagram. Figure 1. Proposed approach to evaluating different compression and error concealment algorithms for videos. 2.2. Brief Review of Video Compression Algorithms Here, we will briefly review video codecs in the market. Wavelet based algorithm J2K for Video J2K is a wavelet based compression algorithm for still images. Motion J2K was developed in 2001. It does not have inter-frame coding and hence the coding efficiency is not as good as other videos codecs such as X-264, X265, etc. DCT based algorithms • JPEG [5] : JPEG is the very first image compression standard. The video counterparts are the MPEG-1 and MPEG-2 standards. • VP8 and VP9 [13] [14] : These video compression algorithms are owned by Google. The performance is somewhat close to X-264. However, it is not as popular as X264. • X-264 [3] : X264 is the current state-of-the-art in video compression. Youtube uses X264. It has good still image compression. • X-265 [4] : This is the next-generation video codec. However, the computational complexity is much more than that of X264. In general, X265 has the same basic structure as previous standards. There are some incremental improvements in X265 as compared to X264, including:  Flexible partitioning  Flexibility in prediction modes and transform block sizes  Sophisticated interpolation and deblocking filters  Sophisticated prediction and signalling of modes and motion vectors X264 and X265 are optimized versions of H264 and H265, respectively. • Daala [9] Recently, there is a parallel activity at xiph.org foundation, which implements a compression codec called Daala [9] . It is based on DCT. There are preand post-filters to increase energy compaction and remove block artifacts. Daala borrows ideas from [8], which was written by one of us (T. Tran). Compression Algorithms (X264, X265, overlapped block transform, JPEG-2000) Images/videos (maritime/sonar) Compression Performance Metrics Communication Channel Decompression and Error Recovery Error Recovery Performance Metrics Received Images/videos (maritime/sonar) Signal & Image Processing: An International Journal (SIPIJ) Vol.10, No.2, April 2019 4 The block-coding framework in Daala can be illustrated in Figure 2. Figure 2. Daala codec for block-based image coding systems. In this study, we have mainly compressed videos using Daala, X264, X265, and JPEG-2000. 2.3. Error Resilient/Error Concealment In [15], several coding schemes were evaluated. From Figure , it can be seen that there is a limit to which the channel errors can be corrected. It should be noted that there are some built-in error concealment mechanisms in codecs such as H264. Flexible Macro Ordering (FMO) is one of them. In one of our papers, we have evaluated some of these mechanisms and found that they are only effect to certain extent [10]. In practical applications, we believe a combination of the built-in error concealment and postdecompression error concealment is needed. Unprotected (SNR 7 dB AWGN) RS+convolutional coded Turbo coded One way decoding Two way decoding Figure 3. Comparison of several error resilient coding schemes [15]. There are still errors that cannot be fixed. Signal & Image Processin 2.4. Principle of Video Error Concealment Similar to still images, error resilient coding incurs more overhead in bandwidth and may not be able to deal with severe channel interferences. It is therefore necessary to apply error concealment techniques to recover lost pixels in the video. Our general approach for these problems is based on the observation that for every small block in a frame, ther e always exist some motion estimations (ME) from the previous or future frames. E ach patch with missing or corrupted values is grouped with similar patches from the partial information of the patch and stacked into columns of a matrix. All the matched motion estimations should have similar underlying image structures and the completed patches should lie in a low dimensional subspace. Therefore, the constructed matrix becomes a very low-rank structure [17]- [26] Figure 4. Bidirectional motion compensation 2.5. Performance Metrics In many papers, researchers used peak signal (SSIM) to algorithm evaluation PSNR or SSIM will be regarded as better algorithms. However, PSNR or SSIM do not correlate well with human perception. Recently, a group of researchers investigated a number of different performance metrics [28]. Extensive experiments were performed to investigate the correlation between human perception with various performance metrics. According to the results found in [28], it was determined that two performance metrics known as human visual system (HVS) and hum an visual system with masking (HVSm) correlate well with human perception. A summary of the findings in [28] can be seen in PSNR-HVSM (HVSm in short) have high correlation with human subjective eva For completeness, we include Table 1. It can be seen that HVSm and HVS have much higher correlation with human perception than PSNR and SSIM in terms of Spearman and Kendall correlation coefficients. Signal & Image Processin g: An International Journal (SIPIJ) Vol.10, No.2, April 2019 Principle of Video Error Concealment [16] Similar to still images, error resilient coding incurs more overhead in bandwidth and may not be to deal with severe channel interferences. It is therefore necessary to apply error concealment techniques to recover lost pixels in the video. Our general approach for these problems is based on the observation that for every small block in e always exist some motion estimations (ME) from the previous or future frames. ach patch with missing or corrupted values is grouped with similar patches from the partial information of the patch and stacked into columns of a matrix. All the matched motion estimations should have similar underlying image structures and the completed patches should lie in a low dimensional subspace. Therefore, the constructed matrix becomes a [26] and can be applied in lowrank techniques for reconstruction. directional motion compensation approach to reconstruct corrupted pixels researchers used peak signal -tonoise ratio (PSNR) or structural similarity algorithm evaluation . Given a fixed compression ratio, algorithms that yield high PSNR or SSIM will be regarded as better algorithms. However, PSNR or SSIM do not correlate well with human perception. Recently, a group of researchers investigated a number of different Extensive experiments were performed to investigate the correlation between human perception with various performance metrics. According to the results found in it was determined that two performance metrics known as human visual system (HVS) and an visual system with masking (HVSm) correlate well with human perception. A summary of can be seen in Figure 5 . It can be seen that PSNRHVS (HVS in short) and M (HVSm in short) have high correlation with human subjective eva For completeness, we include Table 1. It can be seen that HVSm and HVS have much higher correlation with human perception than PSNR and SSIM in terms of Spearman and Kendall g: An International Journal (SIPIJ) Vol.10, No.2, April 2019 5 Similar to still images, error resilient coding incurs more overhead in bandwidth and may not be to deal with severe channel interferences. It is therefore necessary to apply error concealment Our general approach for these problems is based on the observation that for every small block in e always exist some motion estimations (ME) from the previous or future frames. ach patch with missing or corrupted values is grouped with similar patches from the partial information of the patch and stacked into columns of a matrix. All the matched motion estimations should have similar underlying image structures and the completed version of these patches should lie in a low dimensional subspace. Therefore, the constructed matrix becomes a rank techniques for reconstruction. approach to reconstruct corrupted pixels [16]. noise ratio (PSNR) or structural similarity . Given a fixed compression ratio, algorithms that yield high er PSNR or SSIM will be regarded as better algorithms. However, PSNR or SSIM do not correlate well with human perception. Recently, a group of researchers investigated a number of different Extensive experiments were performed to investigate the correlation between human perception with various performance metrics. According to the results found in it was determined that two performance metrics known as human visual system (HVS) and an visual system with masking (HVSm) correlate well with human perception. A summary of HVS (HVS in short) and M (HVSm in short) have high correlation with human subjective eva luation results. For completeness, we include Table 1. It can be seen that HVSm and HVS have much higher correlation with human perception than PSNR and SSIM in terms of Spearman and Kendall Signal & Image Processing: An International Journal (SIPIJ) Vol.10, No.2, April 2019 6 Hence, in addition to PSNR and SSIM, we also used HVS and HVSm for assessing perceptually lossless compression. Figure 5. Comparison of SSIM and HVSM. HVSM has better correlation with human perception [28]. Table 1. Correlation of different metrics to human’s visual perception [28]. 3. E XPERIMENTAL R ESULTS 3.1. Data Our sponsor is interested in low frame rate periscope and sonar videos. We searched the Internet and found some public videos: one periscope video and one sonar video. Signal & Image Processing: An International Journal (SIPIJ) Vol.10, No.2, April 2019 7 3.1.1 Periscope Video The video frame size is 720 x 1280 with 6 fps. There are 42 frames. Two frames are shown below. Figure 6. Two frames of a periscope video. 3.1.2 Sonar Video The sonar video has a size of 720 x 1280 at 6 fps. There are 87 frames. Two frames are shown below. Figure 7. Two frames of a sonar video. 3.2. Video Compression Results 3.2.1 Periscope Video Compression Four compression algorithms (J2K, Daala, X264, X265) were compared. Four performance metrics (PSNR, SSIM, HVS, HVSm) were used for evaluation. As shown in Figure, the performance metrics are very high and can be considered as perceptually lossless. Table summarizes the performance metrics of various codecs at a compression ratio of 0.05. It is worth to mention that X264 performed slightly better than X265. However, X265 performed better at lower compression ratio such as 0.01. J2K and Daala were not as good as X264 and X265. Since the computational burden of X264 is much less than X265, it is suitable for real-time applications. Signal & Image Processin (a) (c) Figure 8. Performance metrics for periscope video compression: (a) Averaged PSNR in dB of all frames versus compression ratio; (b) Averaged SSIM of all frames versus compression ratio; (c) Averaged HVS in dB of all frames versus compression ratio; (d) Averaged HVSm Table 2 . Performance metrics of four codecs at 0.05 compression ratio for a periscope video. Bold numbers J2K X264 X265 Daala 3.2.2 Sonar Video C ompression Sonar videos are somewhat harder to compress as there are a lot noises and textures in the frames. It is well known that noisy frames are hard to compress. Here, we also applied four compression algorithms and four performance metrics to evaluate the resul metrics. It can be seen that all the compression algorithms work well in achieving high numbers. Signal & Image Processin g: An International Journal (SIPIJ) Vol.10, No.2, April 2019 (b) (d) Performance metrics for periscope video compression: (a) Averaged PSNR in dB of all frames versus compression ratio; (b) Averaged SSIM of all frames versus compression ratio; (c) Averaged HVS in dB of all frames versus compression ratio; (d) Averaged HVSm in dB of all frames versus compression ratio. . Performance metrics of four codecs at 0.05 compression ratio for a periscope video. Bold numbers indicate the best performing method. PSNR (dB) SSIM HVS (dB) HVSm (dB) 5 5 . 2 5 0 .83 47.5 50.4 57 0.96 51.5 56.8 56.5 0.9 5 51 55.3 53.75 0.87 46.8 49.8 ompression Sonar videos are somewhat harder to compress as there are a lot noises and textures in the frames. It is well known that noisy frames are hard to compress. Here, we also applied four compression algorithms and four performance metrics to evaluate the resul ts. Figure shows the performance metrics. It can be seen that all the compression algorithms work well in achieving high numbers. g: An International Journal (SIPIJ) Vol.10, No.2, April 2019 8 Performance metrics for periscope video compression: (a) Averaged PSNR in dB of all frames versus compression ratio; (b) Averaged SSIM of all frames versus compression ratio; (c) Averaged HVS in in dB of all frames versus compression . Performance metrics of four codecs at 0.05 compression ratio for a periscope video. Bold numbers Sonar videos are somewhat harder to compress as there are a lot noises and textures in the frames. It is well known that noisy frames are hard to compress. Here, we also applied four compression shows the performance metrics. It can be seen that all the compression algorithms work well in achieving high numbers. Signal & Image Processin Table summarizes the metrics of various codecs at a compression ratio of 0.05. Similar to the periscope video, we also observe that X264 performs better than x265. J2K is the worst case. (a) (c) Figure 9. Performance metrics for sonar video compression: ( compression ratio; (b ) Averaged SSIM of all frames versus compression ratio; ( all frames versus compression ratio; ( Table 3 . Performance metrics of four codecs at 0.05 compression ratio for a sonar video. Bold numbers J2K X264 X265 Daala Signal & Image Processin g: An International Journal (SIPIJ) Vol.10, No.2, April 2019 summarizes the metrics of various codecs at a compression ratio of 0.05. Similar to the periscope video, we also observe that X264 performs better than x265. J2K is the worst (b) (d) Performance metrics for sonar video compression: ( a ) Averaged PSNR in dB of all frames versus ) Averaged SSIM of all frames versus compression ratio; ( c ) Averaged HVS in dB of frames versus compression ratio; ( d ) Averaged HVSm in dB of all frames versus compression ratio. . Performance metrics of four codecs at 0.05 compression ratio for a sonar video. Bold numbers indicate the best performing method. PSNR (dB) SSIM HVS (dB) HVSm (dB) 43 0.785 36.85 39.5 57 .5 0.9 81 5 2 .5 60.5 55 0.98 49.3 56.5 50.6 0. 94 46.25 52.4 g: An International Journal (SIPIJ) Vol.10, No.2, April 2019 9 summarizes the metrics of various codecs at a compression ratio of 0.05. Similar to the periscope video, we also observe that X264 performs better than x265. J2K is the worst in this ) Averaged PSNR in dB of all frames versus ) Averaged HVS in dB of ) Averaged HVSm in dB of all frames versus compression ratio. . Performance metrics of four codecs at 0.05 compression ratio for a sonar video. Bold numbers