Predicting perceptual quality in internet television based on unsupervised learning

symmetry S S Article Predicting Perceptual Quality in Internet Television Based on Unsupervised Learning Jaroslav Frnda 1,* , Jan Nedoma 2, Radek Martinek 3and Michael Fridrich 2 1Department of Quantitative Methods and Economic Informatics, Faculty of Operation and Economics of Transport and Communications, University of Zilina, 010 26 Zilina, Slovakia 2Department of Telecommunications, Faculty of Electrical Engineering and Computer Science, VSB—Technical University of Ostrava, 17. Listopadu 15, 708 33 Ostrava-Poruba, Czech Republic; [email protected] (J.N.); [email protected] (M.F.) 3Department of Cybernetics and Biomedical Engineering, VSB—Technical University of Ostrava, 708 00 Ostrava-Poruba, Czech Republic; [email protected] *Correspondence: jaroslav[email protected] Received: 13 August 2020; Accepted: 16 September 2020; Published: 17 September 2020   Abstract: Quality of service (QoS) and quality of experience (QoE) are two major concepts for the quality evaluation of video services. QoS analyzes the technical performance of a network transmission chain (e.g., utilization or packet loss rate). On the other hand, subjective evaluation (QoE) relies on the observer’s opinion, so it cannot provide output in a form of score immediately (extensive time requirements). Although several well-known methods for objective evaluation exist (trying to adopt psychological principles of the human visual system via mathematical models), each of them has its own rating scale without an existing symmetric conversion to a standardized subjective output like MOS (mean opinion score), typically represented by a five-point rating scale. This makes it difficult for network operators to recognize when they have to apply resource reservation control mechanisms. For this reason, we propose an application (classifier) that derivates the subjective end-user quality perception based on a score of objective assessment and selected parameters of each video sequence. Our model integrates the unique benefits of unsupervised learning and clustering techniques such as overfitting avoidance or small dataset requirements. In fact, most of the published papers are based on regression models or supervised clustering. In this article, we also investigate the possibility of a graphical SOM (self-organizing map) representation called a U-matrix as a feature selection method. Keywords: mapping function; QoE; QoS; self-organizing map; video quality estimation 1. Introduction and Motivation Streaming media has become a popular technology in recent years and represents a major amount of the data delivered via internet protocol (IP)-based networks. Adoption of the next-generation network (NGN) concept allows us to transmit not only data but also voice and video in their digital form via transmission chains originally projected for data transfer (such as e-mail communication, web browsing, etc.). Since multimedia services (especially video) are generally asymmetric (the content is delivered in one direction) real-time critical services, transmission control protocol (TCP) had to be replaced by a connectionless user datagram protocol (UDP) with no guarantee of data delivery (data retransmission causes additional overhead time in the communication process). The UDP avoids the overhead time of the TCP handshake procedure. On the other hand, lost data can create visual impairments during video decoding and playback. Well-known video on demand (VoD) services, e.g., HBO Go, Netflix, and YouTube, still use a reliable connection established by TCP, but are in fact not live stream broadcasting. They use buffers to Symmetry 2020,12, 1535; doi:10.3390/sym12091535 www.mdpi.com/journal/symmetry Symmetry 2020,12, 1535 2 of 16 preload data into reserved area of memory; for instance, YouTube stores approximately 60 s of the encoded video before playback, and Netflix has a 240-s playback buffer. If providers of internet protocol television (IPTV) services want to become successful competitors of traditional terrestrial video broadcasting service companies, they must control and analyze the service quality they promote. Because subjective tests require the participation of many observers, the picture quality is calculated by objective video quality methods. Several well-known methods have been defined for this, such as peak signal to noise ratio (PSNR), video quality metric (VQM), and structural similarity index (SSIM). PSNR is the oldest method but provides fast calculation of results; SSIM and VQM reflect better on end-user subjective video quality perception [ 1 , 2 ]. Generally, subjective evaluation is represented in the form of five-point scale, standardized by the International Telecommunication Union (ITU), called the mean opinion score, where five points stands for the highest perceived quality. Each of these objective metrics uses its own scale. The relationship between the subjective and objective results has not yet been defined exactly, so it is not known how to correctly interpret objective results on the subjective MOS scale. The possibility of using subjective MOS data measured for different QoS scenarios has been explored to come up with a model allowing the prediction of QoE based on QoS results, as well as the derivation of QoS parameters for a given QoE requirement. In order to achieve this, it is essential to understand how measurable parameters reflect the quality of service. QoS and QoE are important for both customers (individuals and businesses) and service providers, and their provision should not only be monitored, but also encouraged and enforced when needed. IPTV, as a part of triple play (a marketing term for offering data, voice, and video services from one service provider), is under the regulations in numerous countries. National regulatory authorities need to prepare qualitative criteria (a set of measured parameters and their limits), as well as to have applications able to measure customer satisfaction levels. QoS regulation should be a part of customer protection, aiming to provide an objective quality of delivered service comparison between providers. In addition, given the importance of better picture quality with IPTV, it is in the company’s best interest to provide a high-quality service; otherwise, the customer will go elsewhere. Our created model takes into account the concepts of both quality evaluation techniques, which results in a functional monitoring system. For that reason, the key objective and motivation behind this work is to combine the results provided by the objective and subjective methods. The proposed system will be in a form of a so-called passive monitoring (nonintrusive) tool that does not affect the inspected transmission chain (e.g., packets manipulation) and will be based on the machine learning algorithm. Machine learning offers data analysis that automates prediction model building. A selected algorithm can organize large, complex datasets, and the training procedure is very fast in comparison to traditional solutions based on back-propagation neural networks. We also tried to inspect the clustering ability of algorithm for feature selection. A Kohonen map allows us to identify and select important variables in the feature spaces, which is another benefit of the clustering technique, but only a few studies have contributed to this knowledge. Our application can be operated as an end-user video quality perception estimator and can help with data flow prioritization settings in content delivery networks. 2. Related Works Several studies propose mapping functions to translate objective results into a subjective point of view. The first endeavor to apply machine learning to this research issue was called pseudo-subjective quality assessment (PSQA) [ 3 ]. Mohamed and Rubino worked with parameters such as packet loss or bitrate as an input to neural network modelling, followed by the objective score, to compute a subjectively perceived quality. They used a very old (nowadays) codec, MPEG 2, altogether with low resolution (352 × 288) and a small sample size dataset, but their article still serves as a basic reference for this work. Symmetry 2020,12, 1535 3 of 16 Valderrama and G ó mez [ 4 ] chose a different set of inputs, including different lengths of the group of pictures (GOP), two prioritization techniques (DiffServ or BestEffort), and bandwidth bottlenecks in the experimental network. They obtained a Pearson correlation coefficient (PCC) slightly above 0.9, but only one resolution (740 × 480) and a high packet loss rate were used. Ma et al. [ 5 ] prepared an error sensitivity model based on spatial and temporal features to estimate the robustness of videos to different packet loss scenarios. The paper by Søgaard et al. [ 6 ] suggested a regression equation for video quality estimation with a PCC oscillating between 0.7 to 0.9, based on the video content type. The main benefit of the paper by Loh and Bong [ 7 ] is the enhancement of the SSIM index. They incorporated the idea of spatial and temporal scene characteristics into the SSIM methodology. They obtained an improvement in precision but, on the other hand, the computational time of their metric doubled compared to the “baseline” performance of SSIM. The successor of video codec H.264, namely H.265/HEVC (high-efficiency video coding), but without ultra-high definition (UHD) sequences included in the testing dataset, was taken into account to create a regression for prediction of the subjective score in [ 8 , 9 ] with an obtained PCC of 0.92. Mustafa and Hameed [ 10 ] had many testing scenarios (packet loss rate, various bitrates and scenes) but worked only with low resolution for video encoding (H.264 codec). They applied a new methodology to numerous machine learning applications (neural network, naïve Bayes, or decision tree) and obtained a classification rate ranging from 0.86 to 0.88. The article by Akhtar et al. [ 11 ] summarizes the recent innovations in this research topic and studies several approaches such as linear and nonlinear associations between the QoE and QoS parameters. Bampis and Bovik [ 12 ] used temporal and spatial information indexes as inputs into the model developing process and gained a high level of model veracity of about 0.9 (PCC). The article by Gu et al. [ 13 ] dealt with the no-reference estimation model. They used input vector elements such as contrast, sharpness, and brightness. They verified their model on six video databases and obtained a classification rate oscillating between 0.73 and 0.9 according to the particular test database. The benefits and drawbacks of backpropagation neural network usage for video quality estimation are described in [ 14 ]. The authors performed several test scenarios and put forward the improvement schemes of a neural network. They prepared a selection of important video sequence characteristics and analyzed their impact on quality prediction. The proposed model operated with a PCC of about 0.91. The authors of this proposed article also contributed to this research. We designed a hybrid method for IPTV quality evaluation based on a backpropagation neural network [ 2 ]. Our application takes content type, bitrate, packet loss rate, and resolution into consideration as input elements, followed by the popular video formats (H.264 and H.265/HEVC). At the time of writing, our model can estimate the end-user perception of picture quality for both video profiles concurrently, which is something that none of the state-of-the-art models do. All papers described in this section tried to develop a model for solving pattern recognition problems. The authors wanted to find regularities (patterns) in data obtained from subjective testing and qualitative parameters of video sequences. This study contributes to filling this gap. We chose a self-organizing map due to its ability of learning without a supervision model and clustering. We wanted to inspect the suitability of unsupervised learning and clustering techniques for video quality prediction. As a type of neural network, a Kohonen map offers generalization ability to estimate the data it has not trained on. A Kohonen map is easy to create, and the training process does not require deep knowledge about machine learning. However, service providers need to know if the offered video stream is of sufficient quality or not. Precise information on the MOS rating does not play a major role in network administrator decisions regarding network settings. Our proposed system has all the advantages of our previously published model, namely, almost real-time quality estimation for both of the most popular video codecs. Symmetry 2020,12, 1535 4 of 16 3. Methodology In our last paper related to this research topic [ 2 ], we proposed a distorted video database where each of the test video sequences obtained a score extracted from subjective and objective evaluation procedures. We selected the objective testing methodology SSIM for its good representation of human quality perception. Another benefit of the SSIM metric is its scale range (0–1), which is easy to normalize for successful used in network training. SSIM is a full reference metric, so it requires original undistorted (reference) video sequences for similarity calculation. The final score ranges from 0 (totally different samples) to 1 (two samples that are exactly alike). The video quality investigation process is affected by three components. Luminance land contrast care measured and compared, followed by structural comparison s. The overall index is a multiplicative combination of these components, as depicted in Figure 1. The similarity measure of two sequences (reference xand test sequence y) can be expressed as follows [1,2]: SIM(x,y)=[l(x,y)]α[c(x,y)]β[s(x,y)]γ, (1) where exponents α >0, β >0, and γ >0 measure the weight of each component. The default setting is α=β=γ=1. Symmetry 2020, 12, x FOR PEER REVIEW 4 of 17 3. Methodology In our last paper related to this research topic [2], we proposed a distorted video database where each of the test video sequences obtained a score extracted from subjective and objective evaluation procedures. We selected the objective testing methodology SSIM for its good representation of human quality perception. Another benefit of the SSIM metric is its scale range (0–1), which is easy to normalize for successful used in network training. SSIM is a full reference metric, so it requires original undistorted (reference) video sequences for similarity calculation. The final score ranges from 0 (totally different samples) to 1 (two samples that are exactly alike). The video quality investigation process is affected by three components. Luminance l and contrast c are measured and compared, followed by structural comparison s. The overall index is a multiplicative combination of these components, as depicted in Figure 1. The similarity measure of two sequences (reference x and test sequence y) can be expressed as follows [1,2]: 𝑆𝐼𝑀(𝑥,𝑦)= 󰇟𝑙(𝑥,𝑦)󰇠󰇟𝑐(𝑥,𝑦)󰇠󰇟𝑠(𝑥,𝑦)󰇠, (1) where exponents α > 0, β > 0, and γ > 0 measure the weight of each component. The default setting is α = β = γ = 1. Figure 1. SSIM metric scheme. The absolute category rating (ACR) is an assessment method developed by the International Telecommunication Union (ITU). The test sequences are shown separately, i.e., one at a time. Each test sequence has to be evaluated by real observers. The results are reported as a MOS value (the rating scale is shown in Table 1). Figure 2 shows the whole procedure. As can be seen, the voting limit is approximately 10 s. The ACR method represents the real situation better because end-users cannot compare the received video stream with the original stream made by the content owner (e.g., a TV station). The test environment (lighting conditions, viewing distance) with a 24” Dell P2415Q UHD met the conditions specified in the recommendations [15]. The recommended number of observers is at least 15, but we had 60 viewers who participated in this experiment (between 18 and 40 years old), with men predominating by 39:21. Viewers had a short break every 30 min, and the maximum test session duration was 1.5 h [15]. Table 1. The rating scale of mean opinion score (MOS) [15]. MOS Quality Rating Impairment 5 Best Imperceptible 4 High Perceptible error, not annoying 3 Medium Slightly annoying (visible error) 2 Low Annoying (visible error) 1 Poor Very annoying (visible error) Figure 1. SSIM metric scheme. The absolute category rating (ACR) is an assessment method developed by the International Telecommunication Union (ITU). The test sequences are shown separately, i.e., one at a time. Each test sequence has to be evaluated by real observers. The results are reported as a MOS value (the rating scale is shown in Table 1). Figure 2shows the whole procedure. As can be seen, the voting limit is approximately 10 s. The ACR method represents the real situation better because end-users cannot compare the received video stream with the original stream made by the content owner (e.g., a TV station). The test environment (lighting conditions, viewing distance) with a 24” Dell P2415Q UHD met the conditions specified in the recommendations [ 15 ]. The recommended number of observers is at least 15, but we had 60 viewers who participated in this experiment (between 18 and 40 years old), with men predominating by 39:21. Viewers had a short break every 30 min, and the maximum test session duration was 1.5 h [15]. Table 1. The rating scale of mean opinion score (MOS) [15]. MOS Quality Rating Impairment 5 Best Imperceptible 4 High Perceptible error, not annoying 3 Medium Slightly annoying (visible error) 2 Low Annoying (visible error) 1 Poor Very annoying (visible error) Symmetry 2020,12, 1535 5 of 16 Symmetry 2020, 12, x FOR PEER REVIEW 5 of 17 Figure 2. Absolute category rating (ACR) testing procedure [15]. The video content type can be defined by the temporal (objects motion) and spatial (luminance) information (TI/SI). The recommendation [16] recognizes several types of video content regarding to these two indexes. Our model then estimates the subjective score (MOS) by extraction of information from the situation in the network (packet loss rate), video encoding parameters (i.e., bitrate, codec type) and the content type. The Shanghai Jiao Tong University research group made available their database of uncompressed UHD video sequences that contain various video content types [17]. These video sequences have a duration of 10 s with a frame rate of 30/s. First, selected video sequences were downloaded in UHD resolution (3840 × 2160), 4:2:0 color sampling with 8-bit color depth (YUV format). These settings represent standard TV broadcasting profile. Secondly, all video sequences were encoded to video profiles, namely H.264/AVC and H.265/HEVC, by using the FFmpeg tool version 4.2 (includes x264 and x265 encoders), which allows for modifying bitrates and resolutions. A streaming process was performed by a combination of FFmpeg (as a streaming server) and VLC Player software (version 3.0.6—receiving side). We captured and saved the video stream transmitted via the local network interface using VLC Player. During the streaming procedure, we initially set the packet loss to 0.1% (the application drops randomly selected packets) at the local interface. Then we repeated this step for packet loss in increments of 0.2%, 0.3%, 0.5%, 0.75%, and 1%. The streaming process simulated the RTP/UDP/IP configuration (FFmpeg: -c copy -f mpegts udp://127.0.0.1:1234) with payload encapsulation in MPEGTS (transport stream) format; thus, we fully adopted the principles of IPTV stream transportation over the IP network [17]. The total number of distorted video sequences was 432. The whole process of making the dataset and evaluation performance is depicted in Figure 3. The descriptive characteristics of the chosen scenes are displayed in Figure 4. For more detailed information on the preparation of the testing video sequences, please see our other paper [2]. Figure 3. Dataset making procedure. Digital television terrestrial broadcasting has a theoretical bandwidth limitation of about 31 Mbps per single radio frequency channel (64—Quadrature amplitude modulation and 8 MHz wide channel). As a result of this restriction, the bitrate of one TV signal can oscillate between 5 Mbps (standard quality), 10 Mbps (FullHD), and 15 Mbps (UHD or premium quality). Typically, several TV signals are broadcasted via one radio channel (multiplexing). Figure 2. Absolute category rating (ACR) testing procedure [15]. The video content type can be defined by the temporal (objects motion) and spatial (luminance) information (TI/SI). The recommendation [ 16 ] recognizes several types of video content regarding to these two indexes. Our model then estimates the subjective score (MOS) by extraction of information from the situation in the network (packet loss rate), video encoding parameters (i.e., bitrate, codec type) and the content type. The Shanghai Jiao Tong University research group made available their database of uncompressed UHD video sequences that contain various video content types [ 17 ]. These video sequences have a duration of 10 s with a frame rate of 30/s. First, selected video sequences were downloaded in UHD resolution (3840 × 2160), 4:2:0 color sampling with 8-bit color depth (YUV format). These settings represent standard TV broadcasting profile. Secondly, all video sequences were encoded to video profiles, namely H.264/AVC and H.265/HEVC, by using the FFmpeg tool version 4.2 (includes x264 and x265 encoders), which allows for modifying bitrates and resolutions. A streaming process was performed by a combination of FFmpeg (as a streaming server) and VLC Player software (version 3.0.6—receiving side). We captured and saved the video stream transmitted via the local network interface using VLC Player. During the streaming procedure, we initially set the packet loss to 0.1% (the application drops randomly selected packets) at the local interface. Then we repeated this step for packet loss in increments of 0.2%, 0.3%, 0.5%, 0.75%, and 1%. The streaming process simulated the RTP/UDP/IP configuration (FFmpeg: -c copy -f mpegts udp://127.0.0.1:1234) with payload encapsulation in MPEG-TS (transport stream) format; thus, we fully adopted the principles of IPTV stream transportation over the IP network [ 17 ]. The total number of distorted video sequences was 432. The whole process of making the dataset and evaluation performance is depicted in Figure 3. The descriptive characteristics of the chosen scenes are displayed in Figure 4. For more detailed information on the preparation of the testing video sequences, please see our other paper [2]. Symmetry 2020, 12, x FOR PEER REVIEW 5 of 17 Figure 2. Absolute category rating (ACR) testing procedure [15]. The video content type can be defined by the temporal (objects motion) and spatial (luminance) information (TI/SI). The recommendation [16] recognizes several types of video content regarding to these two indexes. Our model then estimates the subjective score (MOS) by extraction of information from the situation in the network (packet loss rate), video encoding parameters (i.e., bitrate, codec type) and the content type. The Shanghai Jiao Tong University research group made available their database of uncompressed UHD video sequences that contain various video content types [17]. These video sequences have a duration of 10 s with a frame rate of 30/s. First, selected video sequences were downloaded in UHD resolution (3840 × 2160), 4:2:0 color sampling with 8-bit color depth (YUV format). These settings represent standard TV broadcasting profile. Secondly, all video sequences were encoded to video profiles, namely H.264/AVC and H.265/HEVC, by using the FFmpeg tool version 4.2 (includes x264 and x265 encoders), which allows for modifying bitrates and resolutions. A streaming process was performed by a combination of FFmpeg (as a streaming server) and VLC Player software (version 3.0.6—receiving side). We captured and saved the video stream transmitted via the local network interface using VLC Player. During the streaming procedure, we initially set the packet loss to 0.1% (the application drops randomly selected packets) at the local interface. Then we repeated this step for packet loss in increments of 0.2%, 0.3%, 0.5%, 0.75%, and 1%. The streaming process simulated the RTP/UDP/IP configuration (FFmpeg: -c copy -f mpegts udp://127.0.0.1:1234) with payload encapsulation in MPEGTS (transport stream) format; thus, we fully adopted the principles of IPTV stream transportation over the IP network [17]. The total number of distorted video sequences was 432. The whole process of making the dataset and evaluation performance is depicted in Figure 3. The descriptive characteristics of the chosen scenes are displayed in Figure 4. For more detailed information on the preparation of the testing video sequences, please see our other paper [2]. Figure 3. Dataset making procedure. Digital television terrestrial broadcasting has a theoretical bandwidth limitation of about 31 Mbps per single radio frequency channel (64—Quadrature amplitude modulation and 8 MHz wide channel). As a result of this restriction, the bitrate of one TV signal can oscillate between 5 Mbps (standard quality), 10 Mbps (FullHD), and 15 Mbps (UHD or premium quality). Typically, several TV signals are broadcasted via one radio channel (multiplexing). Figure 3. Dataset making procedure. Digital television terrestrial broadcasting has a theoretical bandwidth limitation of about 31 Mbps per single radio frequency channel (64—Quadrature amplitude modulation and 8 MHz wide channel). As a result of this restriction, the bitrate of one TV signal can oscillate between 5 Mbps (standard quality), 10 Mbps (FullHD), and 15 Mbps (UHD or premium quality). Typically, several TV signals are broadcasted via one radio channel (multiplexing). Symmetry 2020,12, 1535 6 of 16 Symmetry 2020, 12, x FOR PEER REVIEW 6 of 17 (a) Wood (b) Campfire Party (c) Construction Field (d) Runners Figure 4. Test sequences [2,17] (clockwise): (a) a high-motion scene (fast camera rotation); (b) night scene (people sitting next to a fire); (c) low-motion scene (moving of bulldozer, static background); (d) Shanghai marathon (static shooting). Our previously mentioned paper [2] inspected many video sequence features with potential to be a part of the input dataset. The motion characteristics of a video have an impact on how codec can mask missing data during the video reconstruction. In a monochromatic static part of the frame (e.g., sky or grass), it is easy to calculate the missing blocks for the decoding of the video frame. However, in an action movie, scenes change very often. As a result, missing data cause visible compression artifacts. Codec with a high compression ratio (in our case, H.265) is more sensitive to data losses because each block carries more information than the codec with a lower compression ratio (H.264). The bitrate defines how much visual information is encoded, typically per second. Higher resolution requires a better bitrate than a low-resolution video does, but a very high bitrate (e.g., 15 Mbps) is counterproductive if low resolution is selected (redundant information that cannot improve the visual quality) [2,18]. A complete list of the chosen parameters, as well as the projected model output, is given in Table 2. Elements of the input vector are in bold, while MOS (in italics) characterizes customer subjective opinion. As can be seen in Table 2, resolution and codec type were not included in the set of inputs. We used a feature selection technique that proved that the boldface parameters have a serious effect on cluster making. One of the well-known statistical methods for input vector dimensionality reduction (feature selections) is called principal component analysis (PCA). PCA is an algorithm allowing us to reduce the set of variables in a way that analyses the potential correlation between them. As a result of this method, a set of variables called principal components is provided, where none of these variables are cross-correlated [19–21]. We verified the results of this standard procedure by U-matrix visualization (an additional advantage of Kohonen maps) to find the minimum set of input vector elements. In order to reduce redundancy in the data space (potential correlation), SOM can be used by considering the so-called component (or weight) planes. We tested all inputs listed in Table 2; according to the generated planes visualization, we selected four inputs without any visual mutual correlation. We can declare that unselected input aspirants were approximated by the variables presented in bold in Table 2. Figure 4. Test sequences [ 2 , 17 ] (clockwise): ( a ) a high-motion scene (fast camera rotation); ( b ) night scene (people sitting next to a fire); ( c ) low-motion scene (moving of bulldozer, static background); (d) Shanghai marathon (static shooting). Our previously mentioned paper [ 2 ] inspected many video sequence features with potential to be a part of the input dataset. The motion characteristics of a video have an impact on how codec can mask missing data during the video reconstruction. In a monochromatic static part of the frame (e.g., sky or grass), it is easy to calculate the missing blocks for the decoding of the video frame. However, in an action movie, scenes change very often. As a result, missing data cause visible compression artifacts. Codec with a high compression ratio (in our case, H.265) is more sensitive to data losses because each block carries more information than the codec with a lower compression ratio (H.264). The bitrate defines how much visual information is encoded, typically per second. Higher resolution requires a better bitrate than a low-resolution video does, but a very high bitrate (e.g., 15 Mbps) is counterproductive if low resolution is selected (redundant information that cannot improve the visual quality) [2,18]. A complete list of the chosen parameters, as well as the projected model output, is given in Table 2. Elements of the input vector are in bold, while MOS (in italics) characterizes customer subjective opinion. As can be seen in Table 2, resolution and codec type were not included in the set of inputs. We used a feature selection technique that proved that the boldface parameters have a serious effect on cluster making. One of the well-known statistical methods for input vector dimensionality reduction (feature selections) is called principal component analysis (PCA). PCA is an algorithm allowing us to reduce the set of variables in a way that analyses the potential correlation between them. As a result of this method, a set of variables called principal components is provided, where none of these variables are cross-correlated [ 19 – 21 ]. We verified the results of this standard procedure by U-matrix visualization (an additional advantage of Kohonen maps) to find the minimum set of input vector elements. In order to reduce redundancy in the data space (potential correlation), SOM can be used by considering the so-called component (or weight) planes. We tested all inputs listed in Table 2; according to the generated planes visualization, we selected four inputs without any visual mutual correlation. We can declare that unselected input aspirants were approximated by the variables presented in bold in Table 2. Symmetry 2020,12, 1535 7 of 16 Table 2. List of dataset variables. Parameter Description Codec H.264/AVC, H.265/HEVC Bitrate (Mbps) 5, 10, 15 Packet loss rate (%) 0.1, 0.2, 0.3, 0.5, 0.75, 1 Resolution HD, FullHD, UHD a Full reference metric SSIM Content type Static scene, scene with significant motion, night, and sports scene ACR MOS value a HD =high definition (1280 × 720). FullHD and UltraHD denote resolutions of 1920 × 1080 or 3840 × 2160, respectively. The input vector xconsists of the parameters set of segment n, as defined in the example below: xn=              Bitrate Packet loss SSIM Category of scene               =              5 0.1 0.969 1               . (2) The category of the scene parameter is 1 for static, 2 for a night scene, 3 for sport, and 4 for a highly dynamic scene. Nis the number of segments in a session. In this case, the input matrix composed of the vectors of all the segments is as follows: I=[x1,. . . xn,. . . xN]. (3) 3.1. Self-Organizing Map (Kohonen Map) A Kohonen (or self-organizing) map tries to categorize inputs based on their similarity in the input space. This approach differs from learning with a teacher due to its weight modification. A Kohonen map uses competitive learning. Output neurons compete among themselves. The neuron that “wins” this competition (the so-called “winner takes all” neuron) is active and its weight vector is updated (and nodes in its neighborhood, too). SOM can be used for many classification problems because competitive learning makes clusters of neurons (with similar characteristics), while each of the formed groups can be considered as a classification class. A Kohonen map contains only two layers, namely, input and output (competitive) layers. This map allows us to transform a multidimensional dataset into a symmetrical structure of 2D nodes. In the worst-case scenario, this algorithm creates as many classification classes as the competitive layer has nodes. This type of neural network was created by Finnish professor Teuvo Kohonen; therefore, SOM and Kohonen map are interchangeable expressions [ 22 ]. Unsupervised learning does not request target outputs to be included in the dataset; thus, no optimization algorithm (for finding a minimum of the cost function) or linear correlation is presented. Classification rate is the way to evaluate the accuracy of the trained model. We measured the classification rate of different network topologies. The average rate of success actions using SOM should be 85–90% to achieve excellent output veracity. A successfully trained network can predict the response from a set of inputs in the form of an exact position (2D grid xand ycoordinates) of an exciting node within the output layer. We can decide if this node belongs to a particular cluster: cluster labelling depends on its dominant content representation—in our case, the most frequently occurring MOS score within each cluster. Kohonen Map Algorithm First, the weight initialization of each node was performed. The sample input vector was chosen from the training dataset. Then the algorithm tried to find the neuron with the weight vector closest to Symmetry 2020,12, 1535 8 of 16 the input vector. This neuron is called the best matching unit (BMU), and its calculation was based on Euclidean distance (we also tested Manhattan city block distance, but obtained worse results for all tested topologies), as follows [23]: dj=rXN i=0xi(t)−wij(t)2, (4) where dj represents individual elements of the input variables rows, and wij represents the weight between the i-th input and the j-th output node. Then BMU is a node with the minimum Euclidean distance: dj∗=min(dj). (5) Weight adaptation is given by this expression: wij(t+1)=wij(t)+η(t)h(j∗,j)xi(t)−wij(t), (6) where η is a learning rate and h(j∗,j) defines weight adaptation within a certain radius. Each iteration makes the radius of neighborhood function decrease. The basic height of neighborhood function hfor a Kohonen map is: h(j∗,j)=       1, if d(i∗,i)≤r(t) 0, otherwise , (7) where d(i∗,i) stands for the distance between the winning neuron i∗ and specific neuron i , and r is devoted to the radius. The topology may not be circular (grid) only. As depicted in Figure 5, symmetric hexagonal topology is preferred, for instance, by MATLAB software. We chose a batch algorithm for the training procedure. MATLAB creates one batch containing all samples of training data. At the end of the batch (iterative loop), weights are updated and BMU is determined. This approach is much faster in comparison to the sequential mode [24,25]. The last step of the training is the validation of classifier outputs for each tested network topology. U-matrix is a useful method for network outputs visualization. The unified distance matrix (U-matrix) shows the position and size of the grouped nodes, as shown in Figure 6. A graphic analysis is formulated on the Euclidean metric between the inputs and nodes of a competitive layer (MATLAB uses yellow to black scale). Yellow zones can be denoted as clusters and black (dark red) parts as cluster boundaries [ 26 ]. We consider the U-matrix a useful graphical interpretation of the results in case someone attempts to identify similarities in the input dataset and they are not familiar with machine learning applications. Symmetry 2020, 12, x FOR PEER REVIEW 8 of 17 on Euclidean distance (we also tested Manhattan city block distance, but obtained worse results for all tested topologies), as follows [23]: 𝑑=  ∑(𝑥(𝑡)−𝑤(𝑡))   , (4) where 𝑑 represents individual elements of the input variables rows, and 𝑤 represents the weight between the i-th input and the j-th output node. Then BMU is a node with the minimum Euclidean distance: 𝑑∗=𝑚𝑖𝑛(𝑑). (5) Weight adaptation is given by this expression: 𝑤(𝑡+1)=𝑤(𝑡)+𝜂(𝑡)ℎ( 𝑗 ∗, 𝑗 )(𝑥(𝑡)−𝑤(𝑡)), (6) where 𝜂 is a learning rate and ℎ(𝑗∗,𝑗) defines weight adaptation within a certain radius. Each iteration makes the radius of neighborhood function decrease. The basic height of neighborhood function h for a Kohonen map is: ℎ( 𝑗 ∗, 𝑗 )=1, 𝑖𝑓 𝑑(𝑖∗,𝑖)≤𝑟(𝑡) 0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 , (7) where 𝑑(𝑖∗,𝑖) stands for the distance between the winning neuron 𝑖∗ and specific neuron 𝑖, and 𝑟 is devoted to the radius. The topology may not be circular (grid) only. As depicted in Figure 5, symmetric hexagonal topology is preferred, for instance, by MATLAB software. We chose a batch algorithm for the training procedure. MATLAB creates one batch containing all samples of training data. At the end of the batch (iterative loop), weights are updated and BMU is determined. This approach is much faster in comparison to the sequential mode [24,25]. The last step of the training is the validation of classifier outputs for each tested network topology. U-matrix is a useful method for network outputs visualization. The unified distance matrix (U-matrix) shows the position and size of the grouped nodes, as shown in Figure 6. A graphic analysis is formulated on the Euclidean metric between the inputs and nodes of a competitive layer (MATLAB uses yellow to black scale). Yellow zones can be denoted as clusters and black (dark red) parts as cluster boundaries [26]. We consider the U-matrix a useful graphical interpretation of the results in case someone attempts to identify similarities in the input dataset and they are not familiar with machine learning applications. Figure 5. Area of weight modification reducing over time. Figure 5. Area of weight modification reducing over time. Symmetry 2020,12, 1535 9 of 16 Symmetry 2020, 12, x FOR PEER REVIEW 9 of 17 Figure 6. Cluster identification by U-matrix. From the gathered data [2] that is shown in Table 3, we set a SSIM interval range related to each MOS rating. The whole procedure of data preparation and modelling is shown in Figure 7. If the subjective rating is worse than 3 on the MOS scale, the delivered video stream is of poor quality, with considerable artifacts appearing in the image (blurring, ringing artifacts, etc.) The obtained results show that a MOS value of 4 or higher might correlate to a SSIM score of 0.98; a MOS rating between 3 and 4 belongs to the interval 0.95–0.979, and the rest is lower than 3 on the MOS scale. Figure 7. The diagram of self-organizing map (SOM) classifier modelling. Figure 6. Cluster identification by U-matrix. From the gathered data [ 2 ] that is shown in Table 3, we set a SSIM interval range related to each MOS rating. The whole procedure of data preparation and modelling is shown in Figure 7. If the subjective rating is worse than 3 on the MOS scale, the delivered video stream is of poor quality, with considerable artifacts appearing in the image (blurring, ringing artifacts, etc.) Table 3. Extracted structural similarity index (SSIM) scores related to the MOS scale obtained from subjective testing. Data are taken from our previous paper [2]. Video Sequence MOS Intervals (ACR) H.264 (SSIM) H.265 (SSIM) Campfire party ≥4 1–0.95 1–0.98 ≥3<4 0.949–0.92 0.979–0.95 ≥2<3 0.919–0.78 0.949–0.885 Construction field ≥4 1–0.98 1–0.98 ≥3<4 0.979–0.94 0.979–0.96 ≥2<3 0.939–0.865 0.959–0.915 Runners ≥4 1–0.98 N/Aa ≥3<4 0.979–0.94 N/A ≥2<3 0.939–0.86 0.95–0.91 Wood ≥4 1–0.96 N/A ≥3<4 0.959–0.89 N/A ≥2<3 0.889–0.615 0.93–0.725 aN/A=Not Applicable. MOS value was associated with none of the computed SSIM indexes. Symmetry 2020, 12, x FOR PEER REVIEW 9 of 17 Figure 6. Cluster identification by U-matrix. From the gathered data [2] that is shown in Table 3, we set a SSIM interval range related to each MOS rating. The whole procedure of data preparation and modelling is shown in Figure 7. If the subjective rating is worse than 3 on the MOS scale, the delivered video stream is of poor quality, with considerable artifacts appearing in the image (blurring, ringing artifacts, etc.) The obtained results show that a MOS value of 4 or higher might correlate to a SSIM score of 0.98; a MOS rating between 3 and 4 belongs to the interval 0.95–0.979, and the rest is lower than 3 on the MOS scale. Figure 7. The diagram of self-organizing map (SOM) classifier modelling. Figure 7. The diagram of self-organizing map (SOM) classifier modelling. Symmetry 2020,12, 1535 16 of 16 12. Bampis, C.G.; Li, Z.; Bovik, A.C. Spatiotemporal Feature Integration and Model Fusion for Full Reference Video Quality Assessment. IEEE Trans. Circuits Syst. Video Technol. 2019,29, 2256–2270. [CrossRef] 13. Gu, K.; Tao, D.; Qiao, J.; Lin, W. Learning a No-Reference Quality Assessment Model of Enhanced Images with Big Data. IEEE Trans. Neural Netw. Learn. Syst. 2018,29, 1301–1313. [CrossRef] 14. Yuana, Y.; Wang, C. IPTV video quality assessment model based on neural network. J. Vis. Commun. Image Represent. 2019,64, 102629. [CrossRef] 15. International Telecommunications Union. ITU-T P.913. Methods for the Subjective Assessment of Video Quality. Audio Quality and Audiovisual Quality of Internet Video and Distribution Quality Television in Any Environment. 2016. Available online: https://www.itu.int/rec/T-REC-P.913/en (accessed on 12 August 2020). 16. International Telecommunications Union. ITU-T P.910. Subjective Video Quality Assessment Methods for Multimedia Applications. 2008. Available online: https://www.itu.int/rec/T-REC-P.910/en (accessed on 12 August 2020). 17. Song, L.; Tang, X.; Zhang, W.; Yang, X.; Xia, P. The SJTU 4K video sequence dataset. In Proceedings of the 5th International Workshop on Quality of Multimedia Experience (QoMEX), Klagenfurt, Austria, 3–5 July 2013. 18. Paudyal, P.; Battisti, F.; Carli, M. Impact of video content and transmission impairments on quality of experience. Multimed. Tools Appl. 2016,75, 16461–16485. [CrossRef] 19. Cao, L.J.; Chua, K.S.; Chong, W.K.; Lee, H.P.; Gu, Q.M. A comparison of PCA, KPCA and ICA for dimensionality reduction in support vector machine. Neurocomputing 2003,55, 321–336. [CrossRef] 20. Belkin, M.; Niyogi, P. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. 2003,15, 1373–1396. [CrossRef] 21. Mousas, C.; Newbury, P.; Anagnostopoulos, C.N. Evaluating the covariance matrix constraints for data-driven statistical human motion reconstruction. In Proceedings of the 30th Spring Conference on Computer Graphics, Smolenice, Slovakia, 28–30 May 2014; Association for Computing Machinery: New York, NY, USA; pp. 99–106. 22. Kohonen, T. The self-organizing map. Neurocomputing 1998,21, 1–6. [CrossRef] 23. Lendel, V.; Pancikova, L.; Falat, L.; Marcek, D. Intelligent Modelling with Alternative Approach Application of Advanced Artificial Intelligence into Traffic Management. Commun.-Sci. Lett. Univ. Zilina 2017 ,19, 36–42. 24. Xiaofei, Q.; Lin, Y.; Kai, G.; Linru, M.; Tao, F.; Shuangyin, R.; Meng, S. Statistics-Enhanced Direct Batch Growth Self-Organizing Mapping for Efficient DoS Attack Detection. Neurocomputing 2016,182, 66–81. 25. Wehrens, R.; Kruisselbrink, J. Flexible Self-Organizing Maps in kohonen 3.0. J. Stat. Softw. 2018 ,87, 1–18. [CrossRef] 26. Yugendar, P.; Ravishankar, K.R. Analysis of Crowd Flow Parameters Using Artificial Neural Network. Transp. Telecommun. J. 2018,19, 335–345. [CrossRef] 27. Minihi, R.; AlSabbagh, H.M.; Al-Rizzo, H.; Al-Omary, A. End-To-End Throughput for Vanet with and without Cloud Effect. Transp. Telecommun. J. 2019,20, 52–61. [CrossRef] 28. Suying, Y.; Guo, Y.; Chen, Y.; Xie, F.; Yu, C.; Liu, Y. Enabling QoE Learning and Prediction of WebRTC Video Communication in WiFi Networks. 2016. Available online: http://eeweb.poly.edu/faculty/yongliu/docs/ yishuai_icc17.pdf (accessed on 2 September 2020). 29. Peng, B.; Lei, J.; Fu, H.; Shao, L.; Huang, Q. A Recursive Constrained Framework for Unsupervised Video Action Clustering. IEEE Trans. Ind. Inform. 2020,16, 555–565. [CrossRef] 30. Bhat, M.; Thiesse, J.; Le Callet, P. A Case Study of Machine Learning Classifiers for Real-Time Adaptive Resolution Prediction in Video Coding. In Proceedings of the 2020 IEEE International Conference on Multimedia and Expo (ICME), London, UK, 6–10 July 2020; pp. 1–6. 31. Anwar, M.S.; Wang, J.; Khan, W.; Ullah, A.; Ahmad, S.; Fei, Z. Subjective QoE of 360-Degree Virtual Reality Videos and Machine Learning Predictions. IEEE Access. 2020,8, 148084–148099. [CrossRef] 32. Khokhar, M.; Ehlinger, T.; Barakat, C. From Network Traffic Measurements to QoE for Internet Video. In Proceedings of the IFIP Networking Conference, Warsaw, Poland, 20–22 May 2019; pp. 1–9. © 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).