Perceptually Lossless Compression with Error Concealment for Periscope and Sonar Videos

Signal & Image Processing: An International Journal (SIPIJ) Vol.10, No.1, February 2019 DOI : 10.5121/sipij.2019.10101 1 COMPRESSION ALGORITHM SELECTION FOR MULTISPECTRAL MASTCAM IMAGES Chiman Kwan, Jude Larkin, Bence Budavari, and Bryan Chou Applied Research, LLC, Rockville, Maryland, USA ABSTRACT The two mast cameras (Mastcam) onboard the Mars rover, Curiosity, are multispectral imagers with nine bands in each camera. Currently, the images are compressed losslessly using JPEG, which can achieve only two to three times compression. We present a two-step approach to compressing multispectral Mastcam images. First, we propose to apply principal component analysis (PCA) to compress the nine bands into three or six bands. This step optimally compresses the 9-band images through spectral correlation between the bands. Second, several well-known image compression codecs, such as JPEG, JPEG-2000 (J2K), X264, and X265, in the literature are applied to compress the 3-band or 6-band images coming out of PCA. The performance of different algorithms was assessed using four well-known performance metrics. Extensive experiments using actual Mastcam images have been performed to demonstrate the proposed framework. We observed that perceptually lossless compression can be achieved at a 10:1 compression ratio. In particular, the performance gain of an approach using a combination of PCA and X265 is at least 5 dBs in terms peak signal-to-noise ratio (PSNR) at a 10:1 compression ratio over that of JPEG when using our proposed approach. KEYWORDS Perceptually lossless compression; Mastcam images; multispectral images; JPEG; JPEG-2000; X264; X265 1. INTRODUCTION The Mars rover, Curiosity, has many instruments onboard for Mars data collection and in-situ surface characterization [1]. Alpha Particle X-ray Spectrometer (APXS) [2], Laser Induced Breakdown Spectrometer (LIBS) [3][4], and Mastcam [5]-[8] are well-known ones. Quite a few of these instruments are imagers that fight for limited bandwidth to transmit data back to Earth. Currently, the Mastcam images are all compressed using JPEG, a technology developed during the 1990’s [9]. Although JPEG is simple and efficient, the compression ratio can be, at most, between two to three times. There are new compression standards developed in the past two decades. Well-known video codecs include X264 [10] and X265 [11], which are also applicable to still image compression. X264 and X265 also provide lossless compression options. In some applications such as security monitoring in shopping centers or home surveillance, people are still using lossless image compression algorithms (JPEG [9] and J2K [12]) for compressing videos frame by frame. This practice may be too conservative, as lossless compression can only achieve two to three times compression. In an earlier work, we focused on the compression of only the RGB Mastcam images with only three bands [8]. In this research, we aim at achieving perceptually lossless compression with a 10:1 compression ratio for multispectral Mastcam images. We propose to apply a new compression framework for Mastcam images that compresses in both the spectral and spatial domain. First, we propose to apply PCA [13] to compress nine bands into three or six bands in the PCA domain. According to theory, PCA achieves optimal compression in this step. Other spectral Signal & Image Processing: An International Journal (SIPIJ) Vol.10, No.1, February 2019 2 compression techniques such as discrete cosine transform (DCT), fast Fourier transform (FFT), or wavelet transforms are all sub-optimal. Second, well-known codecs (JPEG, X264, X265 and J2K) are then applied to compress the PCA bands. Four performance metrics were used in our study. Two of them are motivated by the human visual system and hence are suitable for judging perceptually lossless image quality. Extensive experiments have been performed using actual Mastcam images. It was observed that the combination of PCA and X265 yielded the best performance. Most importantly, the performance gain is at least 5 dBs in terms PSNR at a 10:1 compression ratio over JPEG when using our proposed approach. It is emphasized here that our paper is an application oriented paper, focusing on a niche application that no one has done a systematic study before. Although some of the compression techniques used in this paper are well-known, we still would like to stress two key contributions of our research. The first key contribution of our project is to integrate PCA and conventional image codecs into a unified framework for compressing Mastcam images. Although PCA has been combined with J2K in [19], this does not mean that the PCA and J2K combination is the best for our application, as every application is different. One should not blindly use the combination of PCA and J2K for a new application without some thorough investigations. It turns out that, based on our extensive studies in this paper, we found that the combination of PCA and X265 is actually the best performing one. The second key contribution is that, to the best of our knowledge, we are the first ones to investigate the possibility of replacing JPEG with some advanced compression schemes for Mastcam images. Based on our investigations so far, it appears that a newer compression framework may need to be used in compressing Mastcam images, as more images can be transmitted with the same bandwidth. Our paper is organized as follows. Section 2 summarizes the technical approach and its components. Section 3 summarizes all the experiments using actual Mastcam images. Finally, concluding remarks will be given in Section 4. 2. TECHNICAL APPROACH 2.1. Proposed Multispectral Image Compression Framework Our overall technical approach can be summarized in Figure 1. First, we apply PCA to compress the 9-band image into three or six bands in the PCA domain. This step is somewhat similar to wavelet or fast Fourier transform (FFT), as the image pixels are transformed to other domains. According to theory, PCA achieves optimal compression as compared to other transformations. Second, we apply some conventional image codecs such as JPEG, J2K, X264, and X265 to compress the three or six PCA bands. After this 2-step compression, the file size will be compared to the original file size to generate the compression ratio. To evaluate the quality of the reconstructed images, an inverse compression process needs to be performed where we first decompress the images using the conventional codecs and then an inverse PCA is performed to reconstruct the nine bands. The reconstructed bands will be compared to the uncompressed nine bands to generate the various performance metrics. Although there are some other compression codecs such as VP8 [14] and VP9 [15] in the literature, we used the most popular ones in the literature where open source codes are available. We briefly describe some of the components shown in Figure 1 below. PCA [13] - PCA has been widely used in many applications, including sensor network data compression and hyperspectral image compression. The idea is to perform a singular value decomposition (SVD) to the original 9-band image. The first Signal & Image Processing: An International Journal (SIPIJ) Vol.10, No.1, February 2019 3 three or six principal components are retained. PCA has the optimal compression performance than others [16]-[21]. JPEG [9] - JPEG is the very first image compression standard which was initially released in 1992. The video counterparts are the MPEG-1 and MPEG-2 standards. X-264 [10] - X264 is the current state-of-the-art in video compression. Youtube, Facebook, etc. all use X264. It also has good performance for still image compression. X-265 [11] - This is the next-generation video codec and has excellent still image compression and video compression. However, the computational complexity is much more than that of X264. J2K [12] - Unlike JPEG, X264, and X265, which all use DCT, J2K is wavelet based [22]-[25]. It was initially released in 2000. Figure 1. Proposed 2-step compression scheme for Mastcam images. 2.2. Performance Metrics In almost all compression systems, researchers used PSNR or structural similarity (SSIM) to evaluate the compression algorithms. Given a fixed compression ratio, algorithms that yield higher PSNR or SSIM will be regarded as better algorithms. However, PSNR or SSIM do not correlate well with human perception. Recently, a group of researchers investigated a number of different performance metrics [26] that were motivated from human visual system (HVS). Extensive experiments were performed to investigate the correlation between human perception with various performance metrics. According to the findings found in [26], it was determined that two performance metrics (HVS and HVS-m) correlate well with human perception. Details can be found in [26]. Hence, in addition to PSNR and SSIM, we also used HVS and HVS-m for assessing perceptually lossless compression with respect to Mastcam images. For completeness, we include a figure from [26] that demonstrates that HVS yields better correlation with human perception. In the past, we have used HVS and HVS-m in several applications [8][27][28]. Signal & Image Processing: An International Journal (SIPIJ) Vol.10, No.1, February 2019 4 Figure 2. Comparison of SSIM and HVS-m. HVS-m has better correlation with human perception [26]. 3. EXPERIMENTAL RESULTS 3.1. Mastcam Imager and Mastcam Images Mastcam imager information is shown in Figure 3 and Table 1. There are six overlapping bands and three non-overlapping bands (L3, L4 and L5 from the left camera and R3, R4, and R5 from the right camera). More details about Mastcam can be found in [1]. Figure 4 and Figure 5 show 16 left Mastcam images and 15 right Mastcam images, respectively. Figure 3. Normalized MSL/Mastcam system-level spectral response profiles for the left eye M-34 camera (top panel) and the right eye M-100 camera (bottom panel) [1]. Table 1. Mastcam filters in the order of increasing wavelengths [1]. Signal & Image Processing: An International Journal (SIPIJ) Vol.10, No.1, February 2019 5 Figure 4. Images from the left Mastcam. Figure 5. Images from the right Mastcam. 3.2. Compression Results For the proposed 2-step approach, we tested 2 different cases: 3-band PCA and 6-band PCA. In both cases, we first applied PCA to the original 9-band Mastcam images and then proceeded to apply the various conventional compression methods. 3.2.1. PCA-3 From Figure 6 and Figure 7, one can see that the J2K, X264, and X265 all perform very closely in all four of the metrics generated for the PCA to 3-band approach. It should be noted that we used compression ratio rather than bits per pixel because we take the image overhead information into account and hence it is more close to actual practice. X265 performs better in all available metrics. X264 also performs closely to the X265 in all metrics around the 0.1-0.16 compression ratios. It even slightly outperformed the X265 in HVS and HVS-m at those compression ratios. It should be emphasized that JPEG is far worse than the other methods. Signal & Image Processing: An International Journal (SIPIJ) Vol.10, No.1, February 2019 6 (a) (b) (c) (d) Figure 6. PCA-3: Comparison of different algorithms for 20 left Mastcam mages shown in Figure 4: (a) PSNR in dB for left images; (b) SSIM for left images; (c) HVS in dB for left images; (d) HVSm in dB for left images. (a) (b) Signal & Image Processing: An International Journal (SIPIJ) Vol.10, No.1, February 2019 7 (c) (d) Figure 7. PCA-3: Comparison of different algorithms for 20 right Mastcam images shown in Figure 5: (a) PSNR in dB for right images; (b) SSIM for right images; (c) HVS in dB for right images; (d) HVSm in dB for right images. 3.2.2. PCA-6 The compression results for PCA-6 are shown in Figure 8 and Figure 9. When applying PCA to generate six bands, the methods had similar relative performance between each other, which is similar to the PCA-3 case. The one noticeable difference between the 6-band and 3-band approach is that X265 and J2K are much closer to each other in all metrics. In the PCA-3 approach excluding the lossless case, X265 outperformed J2K in almost all metrics. For this 6band approach, J2K, however, has slightly better HVS and HVSm at compression ratios of 0.05 to 0.1 in some cases. Similar to the PCA-3 case, JPEG is at least 5 dBs lower than other in terms of PSNR. (a) (b) Signal & Image Processing: An International Journal (SIPIJ) Vol.10, No.1, February 2019 8 (c) (d) Figure 8. PCA-6: Comparison of different algorithms for 20 left images shown in Figure 4: (a) PSNR in dB for left images; (b) SSIM for left images; (c) HVS in dB for left images; (d) HVSm in dB for left images. (a) (b) (c) (d) Figure 9. PCA-6: Comparison of different algorithms for 20 right images shown in Figure 5: (a) PSNR in dB for right images; (b) SSIM for right images; (c) HVS in dB for right images; (d) HVSm in dB for right images. Signal & Image Processing: An International Journal (SIPIJ) Vol.10, No.1, February 2019 9 3.2.3 Comparison of PCA-3 and PCA-6 The PCA-6 case had better results for all methods from 0.1 to 0.2 compression rates corresponding to 10 to 5 times compression, respectively. This can be seen in Figure 10 to Figure 13. At 0.1 compression ratio (Figure 10 and Figure 11), the combinations of PCA-6 and X265, and PCA-6 and J2K have very close performance. For left Mastcam images (Figure 10), the combination of PCA-6 and J2K has better performance in SSIM, HVS, and HVSm. For right images (Figure 11), the results are mixed. The performance of JPEG is the worst. Most importantly, the improvement of the best compression algorithm (X265) is about 5 dBs in terms of PSNR (Figure 10(a) and Figure 11(a)) better than JPEG at 0.1 compression ratio. At 0.2 compression ratio (Figure 12 and Figure 13), the combination of PCA and X265 is overwhelmingly better than other combinations. The improvement of the best compression algorithm (X265) is about 7 dBs in terms of PSNR (Figure 12(a) and Figure 13(a)) better than JPEG at 0.2 compression ratio. Thus, we think that it is about time to recommend to NASA about adopting a new approach to Mastcam image compression. (a) (b) (c) (d) Figure 10. Comparison of PCA-3 and PCA-6 for left Mastcam images. Region of interest: compression ratio of 0.1: (a) PSNR in dB for left images; (b) SSIM for left images; (c) HVS in dB for left images; (d) HVSm in dB for left images.