Classification of ice crystal habits observed from airborne Cloud Particle Imager by deep transfer learning

Xiao, Haixia et al. Article — Published Version Classification of ice crystal habits observed from airborne Cloud Particle Imager by deep transfer learning Earth and Space Science Provided in Cooperation with: Leibniz Institute of Agricultural Development in Transition Economies (IAMO), Halle (Saale) Suggested Citation: Xiao, Haixia et al. (2019) : Classification of ice crystal habits observed from airborne Cloud Particle Imager by deep transfer learning, Earth and Space Science, ISSN 2333-5084, American Geophysical Union, Malden, MA, Vol. 6, Iss. 10, pp. 1877-1886, https://doi.org/10.1029/2019EA000636 , https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2019EA000636 This Version is available at: https://hdl.handle.net/10419/206874 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by-nc-nd/4.0/ Classification of Ice Crystal Habits Observed From Airborne Cloud Particle Imager by Deep Transfer Learning Haixia Xiao 1 , Feng Zhang 1 , Qianshan He 2 , Pu Liu 1 , Fei Yan 3 , Lijuan Miao 4,5 , and Zhipeng Yang 1 1 Key Laboratory of Meteorological Disaster, Ministry of Education (KLME)/Collaborative Innovation Center on Forecast and Evaluation of Meteorological Disaster (CIC‐FEMD), Nanjing University of Information Science and Technology, Nanjing, China, 2 Shanghai Key Laboratory of Meteorology and Health, Shanghai Meteorological Service, Shanghai, China, 3 Hebei Provincial Weather Modification Office, Hebei, China, 4 Collaborative Innovation Center on Forecast and Evaluation of Meteorological Disasters, School of Geographical Science, Nanjing University of Information Science and Technology, Nanjing, China, 5 Department of Structural Development of Farms and Rural Areas, Leibniz Institute of Agricultural Development in Transition Economies, Halle, Germany Abstract Ice clouds are mostly composed of different ice crystal habits. It is of great importance to classify ice crystal habits seeing as they could greatly impact single‐scattering properties of ice crystal particles. The single‐scattering properties play an important role in the study of cloud remote sensing and the Earth's atmospheric radiation budget. However, there are countless ice crystals with different shapes in ice clouds, and the task of empirical classification based on naked‐eye observations is unreliable, time consuming and subjective, which leads to classification results having obvious uncertainties and biases. In this paper, the images of ice crystals observed from airborne Cloud Particle Imager in China are used to establish an ice crystal data set called Ice Crystals Database in China, which consists of 10 habit categories containing over 7,000 images. We propose an automatic classification model of ice crystal habits, called TL‐ResNet152, which is a deep convolutional neural network based on the newly developed method of transfer learning. The results show that the TL‐ResNet152 model could achieve reliable performance in ice crystal habits classification with the accuracy of 96%, which is far more accurate than traditional classification methods. Achieving high‐precision automatic classification of ice crystal habits will help us better understand the radiation characteristics of ice clouds. Plain Language Summary In recent years, Convolutional Neural Networks (CNNs), as one of the representative algorithms used in deep learning, have been widely applied in the field of image classification and have achieved remarkable results. However, to the best of our knowledge, there are few studies regarding the application of deep CNNs to ice crystal image classification. In this paper, we propose an automatic classification model of ice crystal habits called TL‐ResNet152, which is a deep CNN based on a newly developed method of transfer learning. Using the TL‐ResNet152 model, high precision and automatic classification of ice crystal habits could be achieved, furthering our understanding of radiation characteristics of ice clouds. We have set up an ice crystal data set, called Ice Crystals Database in China, which consists of 10 habit categories with 7,282 images. As far as we know, it is the first publicly available ice crystals data set in China, pertaining to ice crystals observed in natural ice clouds. The publication of this database will promote more and more researches into understanding the physical process of ice clouds based on ice crystal habits classification. 1. Introduction Middle and upper tropospheric ice clouds play an important role in the Earth's radiation budget. Various categories of ice crystal habits are associated with ice clouds at different heights. For example, rosette‐shaped ice crystals are usually associated with low, overcast clouds, while column‐shaped ice crystals are usually observed in thin, relatively high clouds (Lawson et al., 2006). Accurate measurements of the size, shape, and concentration of ice particles in these clouds are critical to improving our understanding of the physics of ice cloud formation and the radiative effects of the clouds on global climate change (Baker & Lawson, 2006). Kristjánsson et al. (2000) illustrated that the treatment of ice crystal sizes and habits could have a © 2019 The Authors. This is an open access article under the terms of the Creative Commons Attribution‐NonCommercial‐NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non‐commercial and no modifications or adaptations are made. RESEARCH ARTICLE 10.1029/2019EA000636 Key Points: •An ice crystal data set called Ice Crystals Database in China (ICDC) containing over 7,000 images has been first established •We propose an automatic classification model of ice crystal habits, TL‐ResNet152, which is a convolutional neural network created using transfer learning •The high‐precision automatic classification of ice crystal habits is achieved by using this classification model Correspondence to: F. Zhang, [email protected]n Citation: Xiao, H., Zhang, F., He, Q., Liu, P., Yan, F., Miao, L., & Yang, Z. (2019). Classification of ice crystal habits observed from airborne Cloud Particle Imager by deep transfer learning. Earth and Space Science,6https:// doi.org/10.1029/2019EA000636 Received 20 MAR 2019 Accepted 28 AUG 2019 Accepted article online 1 SEP 2019 XIAO ET AL. 1877 Published online 16 OCT 2019 . 1877 –1886 , significant impact on climate change. In addition, different habits of ice crystals have varying single‐scattering characteristics and correspond to different microphysical properties and radiative properties (Letu et al., 2015, 2019). Moreover, from the perspective of radiative transfer simulations and remote sensing involved in the general circulation models, ice clouds previously could not be reliably represented because the current knowledge on the natural variation of ice crystal shapes is still incomplete (Diedenhoven, 2018). Thus, the classification of ice crystal habits can promote the development of ice cloud remote sensing and radiative transfer simulations. In the previous studies, many researchers analyzed the habits of ice crystal categories. Scoresby (1820) comprehensively classified snowflake morphology and described the shape of columnar and complex forms of ice crystals. The studies showed that the shapes of ice crystals was dependent on the cloud temperature (Magono & Chung, 1966; Pruppacher & Klett, 1997). For example, between −5 and −10 °C, the shapes of ice crystals are hollow columns, solid long needles, sheaths, and scrolls; between −10 and −20 °C, the shapes of ice crystals are plate types, including thick plates of skeleton form, fern‐like crystals with sector‐like branches, stellar crystals, ordinary dendritic, and hexagonal plates (Liou & Yang, 2016). Bailey and Hallett (2004, 2009) also gave a comprehensive description of ice crystal habits for the atmosphere as a function of temperature and ice supersaturation using both laboratory studies and field observations for temperatures from −0to−70 ° C, including more complete ice crystal habit categories. In addition, Walden et al. (2003) used the size attributes of ice crystals to classify them as hollow and solid columns, spheroids, pyramids, plates, and bullet clusters over the Antarctic Plateau in winter. Furthermore, Lawson et al. (2006) collected numerous ice particles observed by a cloud particle imager (CPI) at the South Pole station in summer. They sorted the images of ice crystals into nine habit categories (spheroids and small plates, long columns, thick plates/short columns, plates, rosettes, budding rosettes, bullet rosettes, irregulars, and complex crystals with side planes). Praz et al. (2018) used the method of multinomial logistic regression to classify the six ice crystal habits based on the geometrical and textural descriptors of ice crystals. Although the above studies analyzed the categories of ice crystal habits, a publicly available ice crystal database in China has still not been established. The previous studies of ice crystal habit classification are almost entirely based on the visual distinction of categories by expert experience. There are countless ice crystals in ice clouds with varying shapes, and the task of empirical classification based on naked‐eye observation is unreliable, time consuming, and subjective, which leads to the classification results having clear uncertainties, inconsistancies, and biases. Therefore, an automatic and highly accurate method of ice crystal habit classification is urgently needed. Automatically classifying ice crystal habits in natural ice clouds will be a valuable tool in meteorological analysis. In recent years, as a method of machine learning, deep neural networks have made a series of breakthroughs in the field of remote sensing. Xu et al. (2017) used neural networks to characterize ozone profile shapes. Furthermore, Okamura et al. (2017) studied multipixel retrieval of optical thickness and droplet effective radius of inhomogeneous clouds. LeCun et al. (1998) also proposed LeNet‐5, marking the official launch of the convolutional neural network (CNN), and achieved accurate results in terms of handwriting recognition. CNN was also applied to face recognition and achieved highly accurate research results as well (Taigman et al., 2014). Shi et al. (2017) proposed the use of deep convolutional activations‐based features for ground‐based cloud classification and achieved notable classification results. Zhang et al. (2018) presented an optimized CNN model named CloudNet, which illustrated that CNN could accurately classify various cloud images with different characteristics. These studies show that CNNs have advantages for image classification. However, to the best of our knowledge, there are few studies regarding the application of CNNs to the ice crystal habit classifications. In this paper, we first set up an ice crystal database consisting of 10 habit categories containing more than 7,000 images, and then propose a new method of distinguishing ice crystals by using deep CNNs to achieve an automatic classification of ice crystal habits, which would avoid subjective errors. The rest of the paper is structured as follows. Section 2 describes the data set we used in the experiment. We introduce the basic concept of CNNs and transfer learning, which we used in section 3. The experimental details are discussed in section 4. Section 5 compares the classification performance of several representative deep CNN models, and selects a deep CNN model (TL‐ResNet152) with the best performance in ice crystal habits classification, then gives various evaluation matrices to assess the TL‐ResNet152 model trained from the ice crystal data set. Finally, the conclusions are summarized in section 6. 10.1029/2019EA000636 Earth and Space Science XIAO ET AL. 1878 2. Data The data we used are observed by the CPI on an aircraft (Figure 1), which is named KingAir350. We used Version 2.5 of CPI, which records high‐resolution (2.3‐micron pixel size) digital images of particles that pass through the sample volume at speeds up to 200 m/s. The real‐time image processing crops particle images from the full frame, eliminating blank space and compressing data by >1,000:1. CPI images can be displayed in real time on the aircraft. More information about CPI can be found on this site (http://www.specinc.com/ cloud‐particle‐imager). Owing to aircraft performance and weather conditions, we could not have long‐term observational data, so we collected ice crystals pictures observed by CPI on an aircraft in Hebei, China, during several observation sessions. These ice crystals images with PNG format are observed from 2016 to 2017, specificly on 17 and 18 September and 25 December 2016, as well as 21 February, 22 May, 27 August, and 25 November 2017. To achieve the classification of ice crystal habits, a data set with suitable number of accurately labeled ice crystals is necessary. Although several research groups around the world have done extensive work in analyzing ice crystal images, a public and labeled ice crystal database is not yet reliably available. In this paper, an open and labeled ice crystals database in China is first set up. The newly established ice crystal database is called Ice Crystals Database in China (ICDC). A total of 7,282 unique and representative CPI images of ice crystals are labeled into 10 categories by visual characteristics. The ICDC is constituted by the habits of Budding Rosettes (Bud), Rosettes (Ros), Complex with Sideplanes (Cox), Plates (Plt), Sector Plates (Ser), Long Columns (Loc), Short Columns (Shc), Hollow Columns (Hoc), Spheroids (Sph), and Small Irregulars (Sir). The details of the ICDC are shown in Table 1. The categories of Bud, Ros, and Cox have 1,000 images, Figure 1. Photograph of the Cloud Particle Imager (CPI) under the aircraft wing at the Hebei, China. Table 1 Details of the ICDC Category Number of images Description Bud 1,000 Rosette‐shaped with short branches, the length of branches is usually shorter than 100 μm Ros 1,000 Rosette‐shaped with long branches, the length of branches is usually longer than 100 μm Cox 1,000 Complicated by columns, plates, or other irregular ice crystals, complex structure Plt 640 Having a hexagonal structure and is flat, the surface can be smooth or irregular Ser 87 Having six corners that look like hexagonal snowflakes, and the surface can be smooth or irregular Loc 820 Long and solid column, the length is usually more than 2 times in width Shc 800 Short and solid column, the length is usually less than 2 times in width Hoc 325 Columnar and hollow in the middle Sph 800 Spheroid‐shaped and solid Sir 810 Irregular and particle length is usually smaller than 50 μm Total 7,282 Note. ICDC = Ice Crystals Database in China. 10.1029/2019EA000636 Earth and Space Science XIAO ET AL. 1879 respectively, which is the largest number of images in ICDC. The category with the smallest number of images is Ser, which includes 87 images. Figure 2 shows some representative images of each habit category. 3. Materials and Methods CNN is one of the representative algorithms for deep learning (Hinton et al., 2006; LeCun et al., 2015). Figure 3 shows a structure of deep CNNs including the calculation process, and the main types of layer architecture (LeCun et al., 2015) are Convolution Layer/Conv Layer, Pooling Layer, and Fully Connected Layer/FC Layer. Figure 2. Examples of Cloud Particle Imager images of ice crystals sorted into habit categories. 10.1029/2019EA000636 Earth and Space Science XIAO ET AL. 1880 In recent years, deep CNNs have been widely applied in the image field. For example, the AlexNet model (Krizhevsky et al., 2012) achieves success in the ILSVRC 2012 (ImageNet Large‐Scale Visual Recognition Challenge) by pretraining on the ImageNet database (Deng et al., 2009), which contains over 14 × 10 6 labeled images. However, in the practical application process, it is relatively rare to have a data set of enough labeled images to achieve highly accurate classification, which leads to very few people being able to train entire CNN models from scratch. Besides, it is difficult to guarantee that the model we need is exactly the same as the public pretrained model, and the parameters of the pretrained model indeed help to improve the accuracy of the training in the classification task. To classify ice crystal habits accurately, the most effective way is to use the method of transfer learning, which means that the parameters of public pretrained CNN models are adjusted according to the specific ice crystal habit classification task. In this paper, the deep CNN models of AlexNet, VGGNet (VGG16,VGG19) (Simonyan & Zisserman, 2013), ResNet (ResNet18, ResNet34, ResNet152) (He et al., 2016), SqueezeNet1.0 (Iandola et al., 2016), and DenseNet (DenseNet169, DenseNet201) (Huang et al., 2017), which are pretrained based on ImageNet, and the classic and commonly used models in visual classification, are transferred to the ice crystals images database for training. Suitable CNN models of ice crystal habits classification are set up in this way. 4. Implementation Details To estimate the classification accuracy of deep CNN models based on transfer learning, the ICDC is divided into the training set and the test set with a ratio of 4:1. The data set is verified to make sure that there is no image overlap between the training and test set. Before training, we followed these practices for each image used in the ICDC: the original image is flipped horizontally, and random sampling of 224 × 224 is used, with the per‐pixel mean subtracted (Krizhevsky et al., 2012). Moreover, we adjust the brightness of the images and scale the images to make the experiment more precise. Before the activation process, we adopt batch normalization (Ioffe & Szegedy, 2015). We use stochastic gradient descent with backpropagation (LeCun et al., 1989), with a batch size of 4. To obtain a reasonable weight of the last convolutional layer, our experiments used momentum parameters (Sutskever et al., 2013) with the decay of 0.9, and the learning rate of 0.001. Here, the nine pretrained deep CNN models mentioned above (ResNet18, ResNet34, ResNet152, VGG16, VGG19, SqueezeNet1.0, AlexNet, DenseNet169, and DenseNet201) are used for transfer learning. A pretrained mode is loaded, then the final fully connected layer of the model is reset as 10. Then, the model is trained to 30 epochs. The learning rate is reduced to 0.1 times every seven epochs, and the accuracy rate of test data is estimated after each training. Our experiments use the machine learning software package Pytorch (Paszke et al., 2017), which runs on an NVIDIA GeForce GTX1080Ti with a batch size of 8. Next, we report the results of training and testing from different models and select a model with the highest test accuracy, then save the best model configuration and finally perform extensive evaluations on the test set during the experiment. 5. Experimental Results and Analysis To obtain an accurate model for ice crystal habit classification of ICDC, we compared the performance of the nine models based on transfer learning (hereafter, TL‐ResNet18, TL‐ResNet34, TL‐ResNet152, TL‐VGG16, TL‐VGG19, TL‐SqueezeNet1.0, TL‐AlexNet, TL‐DenseNet169, and TL‐DenseNet201). These models were evaluated by using the accuracy and cross‐entropy loss values during the training and testing stages. Figure 3. The framework of a deep Convolutional Neural Network. 10.1029/2019EA000636 Earth and Space Science XIAO ET AL. 1881 Accuracy is the ratio of the number of samples correctly classified by the model to the total number of ice crystals images for a given training or test data set. If yiis the predicted value of the ith image and y i is the corresponding true value, then the accuracy over N labels is defined as Accuracy y;yðÞ¼1 Nlabels ∑ N i¼0 lyi¼yi ðÞ;(1) where l(x) is the indicator function. If all the ice crystal images' set of predictive labels are consistent with the true set of labels, the accuracy is 1.0. The cross‐entropy loss characterizes the distance between the true distribution and the predicted distribution. Calculated as follows (Boer et al., 2005) Hp;qðÞ¼−∑ N i¼1 px i ðÞlog qx i ðÞðÞ;(2) where p(x) is the sample label in transfer learning, and q(x) is the label of prediction. The smaller the value of the cross‐entropy loss is, the closer the two probability distributions are. The accuracy and cross‐entropy loss values of the nine deep CNN models based on transfer learning during training and testing are shown in Figure 4. As the epoch of training increases, the accuracy of all models increases while the cross‐entropy loss value decreases. Both of them Figure 4. (top row) Accuracy and (bottom row) cross‐entropy loss values of the nine deep Convolutional Neural Network models based on transfer learning during training and testing. Figure 5. The best accuracy of the nine deep Convolutional Neural Network models based on transfer learning in testing. 10.1029/2019EA000636 Earth and Space Science XIAO ET AL. 1882 gradually become stable after about 15 epochs of training. Overall, TL‐AlexNet performs the worst, with the lowest accuracy and the largest cross‐entropy loss values for the stage of training. However, TL‐ResNet152 performs the best, with the highest accuracy and lowest cross‐entropy loss values. For testing, TL‐ SqueezeNet1.0 and TL‐AlexNet have low accuracy and high cross‐entropy loss values, while TL‐ ResNet152 has the highest accuracy and lowest cross‐entropy loss values. Therefore, the TL‐ResNet152 model has the best performance for the classification of ICDC. The best accuracy in the testing process for the nine deep CNN models is shown in Figure 5. The best accuracy rates of TL‐DenseNet169, TL‐ ResNet34, and TL‐ResNet152 are all over 96%, while TL‐ResNet152 has the highest accuracy of 96.57%. The best accuracy of TL‐SqueezeNet1.0 (93.75%) and TL‐AlexNet (94.17%) is significantly lower than other models. According to the above evaluation, the TL‐ResNet152 appears to be the best choice for the classification of ice crystal habits. In this paper, TL‐ResNet152 model is used for the classification of ice crystal habits. To help us better understand the process of the classification for the CNN in TL‐ResNet152 model, the feature maps of TL‐ResNet152 are shown in Figure 6. It can be seen that the shallow layers (conv1‐15, conv2_x‐135, and conv3_x‐500 of TL‐ResNet152) can reappear in the shape of different ice crystals well, while the deep layers (conv4_x‐855 and conv5_x‐1255) display the edge and partial information. The features learned by TL‐ResNet152 exhibit hierarchical characteristics. To evaluate the performance of the TL‐ResNet152 model, we analyze the classification precision, recall, and F‐measure of the individual ice crystals for the 10 habit categories in the ICDC. Precision refers to the fraction of relevant instances among the retrieved instances, while recall is the fraction of relevant instances that have been retrieved over the total amount of relevant instances. A precision/recall reaches its best value at 1 and its Figure 6. Visualization of feature maps in TL‐ResNet152 model, denoted Conv1‐15 (channel number 15 on Conv1), Conv2_x‐135, Conv3_x‐500, Conv4_x‐855, and Conv5_x‐1255. Table 2 Performance Evaluation of the 10 Categories in the ICDC Category P R F 1 ‐measure Bud 0.98 0.95 0.97 Cox 0.98 0.97 0.97 Hoc 0.92 1.00 0.96 Ser 0.86 1.00 0.92 Loc 0.94 0.96 0.95 Plt 0.99 0.95 0.97 Ros 0.96 1.00 0.98 Shc 0.99 0.95 0.97 Sir 0.99 0.93 0.96 Sph 0.93 0.99 0.96 Average 0.95 0.97 0.96 Note. ICDC = Ice Crystals Database in China. 10.1029/2019EA000636 Earth and Space Science XIAO ET AL. 1883 worst score at 0. The precision Pand recall Rare calculated by using true negatives (TN), true positives (TP), false negatives (FN), and false positives (FP) (Zhou,2016). P¼TP TP þFPðÞ R¼TP TP þFNðÞ ;(3) The F 1 ‐measure can be interpreted as an equal‐weighted mean of precision and recall. An F 1 ‐measure reaches its best value at 1 and its worst score at 0. It is expressed as follows: F1¼2PñR PþR;(4) Furthermore, the macroaverage of the precision P, recall R, and F 1 ‐measure can be computed as follows: macro P¼1 N∑ N i¼1 Pi;macro R ¼1 N∑ N i¼1 Ri; macro F1¼2ñmacro P ñmacro R macro Pþmacro R : (5) We use different evaluation matrices for each category in ICDC to evaluate the classification performance of the TL‐ResNet152 model (Table 2). The Ser in Table 2 has a relatively unsatisfactory precision of 86% but has a high recall of 100% and F 1 ‐score of 92%. In addition, the precision, recall, and F 1 ‐measure of other categories in Table 2 are all above 92%, with the average values of 95%, 97%, and 96% respectively, which indicates that the classification result of the TL‐ResNet152 model is effective and extensive for each ice crystal habit category. Table 3 also presents the performance evaluation of TL‐ResNet152 based on the ICDC, using the values of macroaverage. The macroaverage of precision, recall, and F 1 ‐measure is 95.41%, 96.98%, and 96.08%, respectively, and with an average value of 96.16%. The results demonstrate that the TL‐ResNet152 model can achieve near‐perfect classification for the task of ice crystal habits classification. Figure 7 shows the confusion matrix of prediction maps with the ICDC test set. Most types of ice crystal habits are ideally distinguished, with the predicted accuracy of above 96%. The accuracy of Hoc, Ser, and Ros is as high as 100%, which shows that although the number of Ser data set is small, the accuracy is high. Therefore, the small number of Ser data set does not harm the accuracy of the results of the entire data set. It also illustrates that both the data set and the TL‐ResNet152 model are reasonable and accurate and are also superior to the traditional method. Despite this, there is a small number of misclassifications with an error rate ranging from 0.5% to 7.4%. Some incorrect classifications can be explained by morphological and meteorological criterion, for example, TL‐ResNet152 model sometimes incorrectly recognizes Shc as Loc, as they are similar in shape. The model sometimes confuses Sir as Sph, seemingly due to the fact that they are both small in size. These errors can be well understood, because there is a mutual transformation between different categories of ice crystal habit, and an ice crystal particle may have several different shapes over a period of time. 6. Conclusions In this paper, an annotated ice crystal database, known as ICDC, containing 7,282 images from CPI observations on aircraft is established, and we objectively divided the ice crystals into 10 habit categories according to the advice of the meteorologists. By the method of transfer learning, the nine pretrained deep CNN models are used to establish the automatic classification models of ice crystal habits, and then the performance of these deep CNNs is compared. Finally, TL‐ResNet152 model with the best Table 3 Performance Evaluation of TL‐ResNet152 Based on the ICDC macro_P macro_R macro_F 1 Average 0.9541 0.9698 0.9608 0.9616 Note. ICDC = Ice Crystals Database in China. Figure 7. The confusion matrix is used to evaluate the effectiveness of the TL‐ResNet152 model for Ice Crystals Database in China test set. 10.1029/2019EA000636 Earth and Space Science XIAO ET AL. 1884