Information Dissimilarity Measures in Decentralized Knowledge Distillation: A Comparative Analysis

This is a pre-copyedit version of this article. The final authenticated version is available online at: https://doi.org/10.1007/978-3-031-75823-212. Information Dissimilarity Measures in Decentralized Knowledge Distillation: A Comparative Analysis Mbasa Joaquim Molo1,2, Lucia Vadicamo2, Emanuele Carlini2, Claudio Gennaro2, and Richard Connor3 1Department of Computer Science, University of Pisa, Pisa, Italy, [email protected], 2Institute of Information Science and Technologies, CNR, Via G.Moruzzi 1, 56124, Pisa, Italy {lucia.vadicamo,emanuele.carlini,claudio.gennaro}@isti.cnr.it 3School of Computer Science, University of St Andrews, St Andrews, KY16 9SS, Scotland [email protected] Abstract. Knowledge distillation (KD) is a key technique for transferring knowledge from a large, complex “teacher” model to a smaller, more efficient “student” model. Although initially developed for model compression, it has found applications across various domains due to the benefits of its knowledge transfer mechanism. While Cross Entropy (CE) and Kullback-Leibler (KL) are commonly used in KD, this work investigates the applicability of loss functions based on underexplored information dissimilarity measures, such as Triangular Divergence (TD), Structural Entropic Distance (SED), and Jensen-Shannon Divergence (JS), for both independent and identically distributed (iid) and non-iid data distributions. The primary contributions of this study include an empirical evaluation of these dissimilarity measures within a decentralized learning context, i.e., where independent clients collaborate without a central server coordinating the learning process. Additionally, the paper assesses the performance of clients by comparing pairwise distillation averaging among clients to conventional peer-to-peer pairwise distillation. Results indicate that while dissimilarity measures perform comparably in iid settings, non-iid distributions favor SED and JS, which also demonstrated consistent performance across clients. Keywords: Information dissimilarity measure · Divergence Function · Knowledge Distillation · Distributed intelligence. 1 Introduction The integration of Artificial Intelligence in edge processing has led to the emergence of an interdisciplinary field known as Distributed Intelligence or Edge Intelligence, which aims to develop systems composed of software agents, robots, sensors, and computer systems that can collaborate effectively [22,23,15]. In this field, Knowledge distillation (KD) has been employed to facilitate knowledge transfer between edge devices, enhancing the development of more efficient and accurate models [28]. KD is a machine learning technique designed to transfer knowledge from a large, complex model (the teacher) to a smaller, more efficient one (the student) [10,8,7]. In addition to its This is a pre-copyedit version of this article. The final authenticated version is available online at: https://doi.org/10.1007/978-3-031-75823-212. 2 Molo et al. Logits Soft predictions Input data Hard labels (ground truth) Client 1 Hard Predictions Fully-supervised Loss …Softmax (T=t) Softmax (T=1) Distillation Loss Total Loss Client 𝑘"backbone Client 𝑘"classifier Logits Soft predictions Input data Client 𝑘 Hard Predictions Fully-supervised Loss Softmax (T=t) Softmax (T=1) Distillation Loss Total Loss … Logits Soft predictions Input data Client 𝐾 Hard Predictions Fully-supervised Loss Softmax (T=t) Softmax (T=1) Distillation Loss Total Loss … Hard labels (ground truth) Hard labels (ground truth) Fig. 1: KD-based decentralized network consisting of Kclients, where distillation is performed using soft predictions for effective knowledge transfer. primary role in model compression, it has started to find applications in other areas, including distributed intelligence [3] and continual learning [4]. In KD-based distributed learning framework, clients exchange information to enhance their learning process, where each client operates both as learner and source of knowledge for other clients. These clients are part of a decentralized system where no single model acts as the central teacher. Instead, each client trains on its local dataset and shares knowledge with others. As illustrated in Fig. 1, this information exchange is achieved through a combination of two types of losses. The first loss component, indicated as “fully-supervised loss”, is usually the cross-entropy (CE) with “hard” targets derived by the ground-truth labels of the input samples. The second component is the “distillation loss” designed to ensure that each learning client mimics the output of other remote clients [25]. This loss is typically implemented by comparing the probability distributions of the models involved, where one model acts as the student and others take turns serving as teachers. This encourages the student’s output probabilities to closely match those of the teacher. The model’s output probabilities are typically computed using a softmax layer. Adjusting the softmax temperature during training has proven to be crucial in metric learning and distillation processes. In the context of distributed intelligence, this technique is also employed to generate soft predictions for effective distillation. Hence, the distillation loss is expressed as minimizing the gap between the soft predictions of one client with respect to the soft predictions of all other clients [1,2,27]. Given that the softmax function transforms an array of logits into an array of positive values summing to 1, various information dissimilarity measures can theoretically be used to implement the distillation loss. However, in practice, it is predominantly realized using CE, in addition to Kullback-Leibler (KL) Divergence, and Mean Squared Error (MSE) [13]. These methods have been extensively studied and proven effective for knowledge transfer in diverse machine learning tasks, while a wide range of information distance functions remain unexplored in the literature related to distributed learning. This is a pre-copyedit version of this article. The final authenticated version is available online at: https://doi.org/10.1007/978-3-031-75823-212. Information Dissimilarity Measures in Decentralized Knowledge Distillation 3 In this work, we break new ground by investigating alternative dissimilarity measures – specifically, Triangular Divergence (TD), Structural Entropic Distance (SED), and Jensen-Shannon (JS) divergence – in the context of KD for decentralized learning scenarios. Recently, the correlations among these measures and the commonly used CE have been examined in [6] for independent and identically distributed (iid) data. Our work aims to expand the understanding of how these dissimilarity measures can enhance KD techniques, particularly in settings where data distribution may vary across learning clients (with a non-iid data distribution). To the best of our knowledge, our study is the first to empirically evaluate the effectiveness of TD, SED, and JSD for KD in a decentralized learning framework, offering novel insights and expanding the potential of KD applications beyond conventional CE and KL-based approaches. Our main contributions include designing a distributed KD environment suitable for investigating the aforementioned information dissimilarity measures and examining the performance of a set of clients by comparing pairwise distillation averaging among clients to the conventional peer-to-peer pairwise distillation, considering the various information dissimilarity measures. The rest of this article is structured as follows. Section 2 provides background and related works on knowledge distillation and the dissimilarity measures utilized. Section 3 details the fully decentralized learning model employed in our study. Section 4 presents our experimental setup, while Section 5 discusses the results. Section 6 provides the conclusions. 2 Background and Related Works 2.1 Information Dissimilarity Measures and Statistical Divergences Information distance refers to a measure that quantifies the dissimilarity between two sources of information (e.g., two finite objects). This concept is distinct but also related to statistical divergences, which quantify the dissimilarity between two probability distributions. For example, some information distances, including SED, can be used to compare also probabilities, while statistical divergence can be interpreted as information distances when the source of information are probability distributions. In the following, we provide formal definitions of the divergence functions and information distances used in this paper4. Kullback-Leibler Divergence. The KL divergence measures the difference between two probability distributions as the amount of information lost when one distribution is used to approximate the other. Given two distributions qand p, it is defined as KL(q:p) = N ∑ i=1 qilog qi pi (1) 4Please note that, as in [6], we use the delimiter ‘:’ as argument separator of non-symmetric divergence instead of the double bar notation ‘||’ used in information theory. This is a pre-copyedit version of this article. The final authenticated version is available online at: https://doi.org/10.1007/978-3-031-75823-212. 4 Molo et al. Jensen-Shannon Divergence. The JS divergence, historically introduced in [26], is a “smoothed, symmetrised“ version of KL divergence and can be interpreted as the total KL divergence relative to the average distribution q+p 2[21]. In this paper, we use the following definition: JS(q,p) = 1 2KLq:q+p 2+KLp:q+p 2 (2) Structural Entropic Distance. SED [20] is an information-theoretic measure that compares the Shannon entropy Hof two probability vectors with that of their arithmetic mean, where H(p) = −∑N ipiln pirepresents the amount of information needed to describe the probability vector p= [p1,...,pN][5]. Considering two probability vectors pand q, SED can be calculated as the ratio of the complexity of the mean vector to the geometric mean of the complexities of individual vectors: SED(q,p) = C(q+p 2) pC(q)C(p)−1 (3) where the complexity is computed as C(p) = b−∑N i=1pilogbpi. The formulation in Eq. (3) gives an outcome in the range [0,1], where 0 implies the two input vectors are identical, and 1 implies that they are orthogonal. Triangular Divergence. The Triangular Divergence 5, also known as Triangular Discrimination [24] is defined as: TD(q,p) = ∑N i=1 (qi−pi)2 qi+pi. Since the range of this function is [0,2], in our work we use its scaled form: TD(q,p) = 1 2 N ∑ i=1 (qi−pi)2 qi+pi =1− N ∑ i=1 2qipi qi+pi (4) where the formulation in the right part of Eq. (4) is an optimized version obtained observing that (qi−pi)2= (qi+pi)2−4qipiand ∑N i=1pi=∑N i=1qi=1. Cross Entropy. CE is a divergence measure widely used in machine learning to compare two probability distributions. It is defined as: CE(q:p) = − n ∑ i=1 qilog pi(5) It is worth noting that in the context of machine learning, as shown in [6], for spaces with certain properties, CE, KL, JS, TD shows very tight correlation. Specifically, if q is fixed, the perfect correlation between cross-entropy and Kullback-Leibler divergence is well-known and derives from simple algebra (KL(q:p) = CE(q:p)−H(q)) [6]. 5Note that its square root is a metric, referred to as Triangular Distance,Vincze-Le Cam distance and the symmetric chi-squared distance [17] This is a pre-copyedit version of this article. The final authenticated version is available online at: https://doi.org/10.1007/978-3-031-75823-212. Information Dissimilarity Measures in Decentralized Knowledge Distillation 5 Moreover, Jensen-Shannon correlates almost perfectly with triangular divergence in almost all high-dimensional spaces [24], while cross-entropy and triangular divergence are strongly correlated when the probabilities are obtained within the softmax function (Eq. (6)) with high temperature. Note that triangular is much cheaper calculation than cross-entropy and if the correlation is very strong the latter may be used instead. 2.2 Knowledge Distillation as the Teacher-Student Approach Knowledge distillation was initially introduced to transfer knowledge from pre-trained teacher (large) networks to student (small) networks. This involves approximating the soft output or intermediate representation of teacher networks, aiming to derive a compact and faster model [16]. Concretely, for any input data x, the teacher network generates a vector of logits z(x)=[z1(x),...,zN(x)] that are turned into a probability vector p(x)=[p1(x),...,pN(x)] using the softmax function: pi(x) = ezi(x) ∑N jezi(x). Typically, neural networks produce probability distributions with sharp peaks, which might lack informativeness. To address this, Hinton et al. [10] proposed temperature scaling in the softmax to soften these probabilities: pi(x,T) = ezi(x)/T ∑N jezj(x)/T,∀i∈ {1,...,N}(6) where Tis a hyperparameter called temperature. In KD, both the student and the teacher generate softened probability distributions, denoted as pS(x,T)and pT(x,T), respectively. The student’s total loss is then defined as a linear combination of a supervised student loss Lstu and a knowledge distillation loss LKD: L=αLstu +(1−α)LKD (7) where α∈[0,1]is a hyperparemeter. Typically, Lstu =CE (y:pS(x,T=1)) and LKD = CE (pT(x,T=t):pS(x,T=t)), with ybeing the hard labels (ground-truth). Note that the distillation loss is expressed as minimizing the gap between the output representation of the teacher and the output representation of the student. KD-based Distributed Learning. Recent research has explored KD for decentralized learning [28]. While much of this work focuses on a central teacher supervising student model training, there is a growing interest in fully decentralized settings where multiple clients collaborate to share knowledge without relying on a central authority. Kim et al. [13] explored the role of the temperature hyperparameter in KD, showing higher temperature results in logit matching, which generally offers better generalization than label matching obtained with lower temperatures. They proposed employing MSE loss for direct logit matching. They showed that KL divergence loss stretches the second-to-last layer representations more than MSE loss and that KL divergence, especially with low temperature, is more resilient to noisy labels. Mishra et al. [18] developed EarlyLight, a method for training lightweight deep neural networks (DNNs) on edge devices using knowledge distillation from larger DNNs, considering also factors This is a pre-copyedit version of this article. The final authenticated version is available online at: https://doi.org/10.1007/978-3-031-75823-212. 6 Molo et al. Table 1: Summary of notation used Notation Description TTemperature in the softmax NNumber of classes (G,ε)Network of clients. Gis the set of nodes, εis the set of edges K,kNumber of clients, Index of current client CkCurrent client Dk= (Xk,yk)Local annotated dataset on client k.Xkis the data, ykare the labels (x,y)∈DkData sample xand the corresponding label y Φk,φSet of indices of remote clients with respect to Ck, Index of a remote client Mk= [Mk h1,Mk h2]Multi-head model held by client k wk= [wk 1,wk 2]Weight parameters of the local model of client k Lk,CE Fully supervised Loss computed on client k Lk,KD Distillation loss used for client k αloss weight parameter like storage, processing speed, and execution time. Molo et al. [19] proposed a knowledge distillation approach for vehicle detection using smart cameras in parking lots, where a large detector (teacher) guides smaller edge-based models (students) without additional labeled data. Their experimental results showed that students improve performance and can even surpass models trained with annotations. Other approaches used a KD-based learning without a single teacher. Zhmoginov et al. [28] introduced Multi-Headed Distillation for distributed learning on the ImageNet dataset. This approach uses multiple model heads distilling to each other and simultaneous distillation of client model predictions and network embeddings, resulting in significantly higher accuracy than naive distillation methods. Jin et al. [12] introduced a personalized Federated Learning (FL) framework using self-KD to transfer historical personalized knowledge, balancing personalization and generalization. Similarly, Jeong et al. [11] addressed personalization challenges in FL for clients with diverse data and behaviors by proposing a KD-based algorithm to compare local models, enhancing client performance without data sharing and showing improved test accuracy, especially under non-iid data distributions. Most works in the literature use KL divergence or CE as dissimilarity measures for distillation. However, there remains significant potential to investigate and utilize alternative dissimilarity measures, which could offer new insights into efficient knowledge transfer and performance across various learning tasks and scenarios. 3 Fully Decentralized Learning Model In this section, we outline the decentralized learning environment used to evaluate the effectiveness of various information dissimilarity measures, introduced in Section 2.1, whose results are discussed in Section 5. The notation used is summarized in Table 1. We consider a full network of Kclients represented by a directed graph (G,ε), where G={Gk|k∈K}is a set of nodes and εis the set of edges between the nodes. This is a pre-copyedit version of this article. The final authenticated version is available online at: https://doi.org/10.1007/978-3-031-75823-212. Information Dissimilarity Measures in Decentralized Knowledge Distillation 7 Client 1 Client 𝐾 Client 𝑘 𝓜𝒉𝟏 𝒌 𝓜𝒉𝟐 𝒌 𝓜𝒉𝟐 𝟏 𝓜𝒉𝟏 𝟏 𝓜𝒉𝟏 𝑲 𝓜𝒉𝟐 𝑲 Distillation with 𝓜𝒉𝟏 𝑲 Distillation with 𝓜𝒉𝟏 𝒌 Distillation with 𝓜𝒉𝟏 𝑲 Distillation with 𝓜𝒉𝟏 𝟏 Distillation with 𝓜𝒉𝟏 𝟏 Distillation with 𝓜𝒉𝟏 𝒌 Fig. 2: KD-based decentralized network consisting of Kclients, where distillation is performed using soft labels for effective knowledge transfer. In this setting, the first head of each client is communicated to the neighboring clients. Each node Gkrepresents a client Ckholding a local dataset Dkcomposed of a pair (Xk,yk), with Xk={xk i}I i=1representing the set of input data and yk={yk i}I i=1the corresponding ground-truth labels. Each client Ckholds a model Mk, which we assume to be a multi-head neural network. Specifically, the model has a backbone, which is the main body of the neural network that processes input data into a feature representation, and two heads, which take the features extracted by the backbone and perform final task-specific operations. The heads consist of a set of fully connected layers added on top of the backbone. We denote the models consisting of the backbone and the first head as Mk h1, and the backbone and the second head as Mk h2. The model with the first head, Mk h1, is trained on the local distribution Dk, while the second, Mk h2, is trained using knowledge distillation from connected clients. The considered KD-based training procedure for this decentralized network involves training multiple clients concurrently, allowing them to share knowledge through distillation to improve overall model performance. Initially, each client’s first model Mk h1is trained in a supervised manner until convergence with local data Dk. Then, as shown in Fig. 2, for each k∈ {1,...,K}, the first model from client Ckis shared with all outgoing connected clients in G. Concurrently, client Ckreceives the first head from all other incoming connected clients. This exchange enables each client to integrate knowledge from others while preserving their local data and model specialization, facilitating collaborative learning across the decentralized network. For the purposes of this study, we assume that all clients are interconnected. However, the proposed approach can be easily adapted to accommodate networks with different topologies and size. For a fixed k, we used the notation Φkto indicate all the indices except k. We refer Ck as the current client and {Cφ|φ∈Φk}as the remote clients. So, once the first head of the models are trained, Ckcommunicates Mk h1to all remote clients and receives the models This is a pre-copyedit version of this article. The final authenticated version is available online at: https://doi.org/10.1007/978-3-031-75823-212. 8 Molo et al. {Mφ h1}φ∈Φkfrom them. The client Ckperforms distillation using the available models from remote clients to train its second head Mk h2. Specifically the parameters wk 2of Mk h2 are trained by optimizing a local total loss Lk, which is obtained as a combination of a cross-entropy loss Lk,CE and a distillation loss Lk,KD: Lk=αLk,CE +(1−α)Lk,KD,(8) where α∈[0,1]is a parameter that weights the contribution of the losses with respect to the total loss. This dual-phase training approach allows each client to effectively train its local model while leveraging shared knowledge from other clients, improving generalization and performance across the network. The cross-entropy loss Lk,CE =E(x,y)∼DkLCE (wk 2,x,y)(9) is used to minimize local prediction with respect to the ground-truth labels of local data.6 For defining the distillation loss Lk,KD we considered two alternatives: – Case 1: The sum of pairwise dissimilarities between the current client’s soft-prediction and remote client’s soft-predictions. – Case 2: A distillation loss based on the dissimilarity between the current client’s soft-predictions and the average of soft-predictions from remote clients. Formally, let pk(wk 2,x) = [pk 1,pk 2,...,pk N]denote the softmax output obtained using the Mk h2model for the input data x, and pφ(x)=[pφ 1,pφ 2,...,pφ N]the softmax outputs of a remote client φfor the input data x(obtained using the pre-trained Mφ h1model). For Case 1, we used Lk,KD(wk 2,x) = ∑ φ∈Φk Ex∼Xkfpk(wk 2,x),pφ(x)(10) where fcan be any divergence measure (e.g., CE,KL, TD, SED, JS). Since the sum of pairwise dissimilarities is used, we refer to this case as "sum" in the experiments. For Case 2, referred to as "average" in the experiments, we used Lk,KD(wk 2,x) = Ex∼Xkf pk(wk 2,x),∑φ∈Φkpφ(x) |Φ|!(11) Algorithm 1 summarizes the considered distillation training procedures. 4 Experimental Setup Our analysis was conducted on a decentralized network consisting of three interconnected clients. This topology serves as a baseline evaluation, with plans for future work 6Please note that LCE (wk 2,x,y)is simply the CE dissimilarity (Eq. (5)) between the output of Mk h2model for the input xand the true labels y This is a pre-copyedit version of this article. The final authenticated version is available online at: https://doi.org/10.1007/978-3-031-75823-212. Information Dissimilarity Measures in Decentralized Knowledge Distillation 9 Algorithm 1: Decentralized Training with Knowledge Distillation Data: (G,ε)graph representing a network of Kclient {C1,...CK} Local datasets Dk= (Xk,yk), where Xk={xk i}I i=1is set of input data and yk={yk i}I i=1 is the set of labels associated with each input, for all k∈ {1,...,K}. Result: Trained model parameters for each client. // Initialization foreach client k ∈ {1,...,K}in parallel do /* Train the model with the first head, Mk h1, until convergence. wk 1are the model parameters to be updated */ Mk h1←LocalModelTraining LCE (wk 1,Dk) // Inizialize the model with the second head, Mk h2. backbone(Mk h2)←backbone(Mk h1) head(Mk h2)randomly inizialized end // Communication for each client k ∈ {1,...,K}in parallel do Φk←indices of incoming connected clients in G// remote client indices Share Mk h1with all outgoing connected clients in G Receive Mφ h1from all remote clients φ∈Φk end // Knowledge Distillation foreach client k ∈ {1,...,k}in parallel do /* Train the model with the second head Mk h2using KD until convergence. Use the loss Lk=αLk,CE +(1−α)Lk,KD, where wk 2 are the model parameters to be updated, Lk,KD is calculated either using Eq. Eq.10 or 11 */ Mk h2←LocalModelTrainingLk(wk 2,Dk) end to extend the analysis to networks with more clients and various connectivity topologies. We studied the effectiveness of different information dissimilarity measures (namely, CE, KL, SED, TD, JS) on distributed learning systems with different levels of data heterogeneity, ranging from scenarios where the data distribution is uniform across all clients (iid) to more extreme situations where each client focuses on its own specific tasks (non-iid). For this purpose, we used the CIFAR-10 [14] dataset and the SUN397 [29] dataset. We split the datasets into three subsets, corresponding to three clients in total. For the CIFAR-10, the iid distribution is obtained by shuffling and evenly splitting the entire dataset, ensuring each client has different samples. For the non-iid distribution across the clients, we followed the configuration in [28]. Each client Ckreceives a subset {ℓi}of the labels, which are designated as primary labels for Ck. Labels not included in {ℓi}are considered secondary for Ck. Samples for each label ℓare distributed randomly among clients, with a higher probability (1 + γtimes greater) of being assigned to