Review on deep learning optimization using knowledge and dataset distillation in medical imaging diagnostics

Review on deep learning optimization using knowledge and dataset distillation in medical imaging diagnostics Nor-Elhouda Laribi1, Djamel Gaceb2, Abdellah Rezoug3, and Fay¸cal Touazi4 1LIMOSE Laboratory University M’Hamed Bougara Boumerdes, Algeria, [email protected] 2LIMOSE Laboratory University M’Hamed Bougara Boumerdes, Algeria, [email protected] 3LIMOSE Laboratory University M’Hamed Bougara Boumerdes, Algeria, [email protected] 4LIMOSE Laboratory University M’Hamed Bougara Boumerdes, Algeria, [email protected] Abstract The integration of deep learning-based artificial intelligence solutions in hospital environments introduces significant challenges, including data privacy restrictions, limited computational resources, and constraints related to the quality and simplicity of the models used. In this review, we highlight the recent advancements in knowledge distillation and dataset distillation as emerging solutions to these challenges in the field of medical imaging. These techniques offer practical benefits in clinical settings by enabling faster training, reduced model size, improved inference speed, and enhanced accuracy, while supporting privacy-preserving learning across decentralized systems and edge devices. Knowledge distillation transfers knowledge from a complex to a simple model, enabling efficient deployment without high loss in diagnostic performance. Dataset distillation, by contrast, focuses on synthesizing datasets that match the pretrained model on real data, reducing data storage requirements. Together, these methods improve learning efficiency, model accuracy, and resource optimization in hospital workflows. However, their integration into medical environments also presents limitations. Challenges such as pipeline complexity, scalability issues, and performance inconsistency across architectures or high-resolution tasks still persist. Overall, this review provides a comprehensive overview of potential and limitations of these two types of distillations in healthcare, offering insights into how these methods can support more scalable, accurate, and privacy-aware AI solutions for medical imaging. Keywords: Healthcare, medical imaging, deep learning, knowledge distillation, dataset distillation, data privacy. 1 Introduction Artificial intelligence (AI), and deep learning in particular, has become increasingly crucial in medical imaging, offering significant improvements in diagnostic accuracy, efficiency, and decision support systems. From radiology to pathology, deep learning models have demonstrated capabilities that rival or exceed human experts in specific tasks. However, deploying these powerful models in real-world hospital settings presents significant challenges. The clinical environment presents challenges, including stringent data privacy, limited computational resources in edge settings, and the practical need for real-time or near-real-time inference. These constraints pose significant challenges to the adoption of conventional, large-scale deep learning models, which are often data-intensive, resource-requirement. To address these limitations, two emerging strategies have gained traction in the research community: Knowledge Distillation (KD) and Dataset Distillation (DD). These techniques aim to retain the performance benefits of deep learning while reducing the computational and data demands that often hinder clinical deployment. Knowledge distillation works by transferring knowledge from a large, complex model (the ”teacher”) to a smaller, more efficient one (the ”student”), preserving accuracy while improving speed and reducing resource usage. Dataset distillation, on the other hand, generates compact synthetic datasets that can replicate the behavior of real data, reducing storage needs and enabling faster training cycles and deployment settings. In this review, we provide a comprehensive overview of both KD and DD techniques in the context of medical imaging, with a particular focus on their application to brain MRI-based disease 48 diagnosis. We explore the methodological foundations of these approaches, analyze recent advancements, and examine their limitations in clinical settings. Through comparative studies, we highlight how various KD strategies, such as soft label supervision, intermediate feature transfer, dual-stream learning, and attention-based mechanisms, enable the compression of large teacher models into lightweight student models without significant loss in performance. In parallel, DD methods have demonstrated the ability to achieve competitive results across a range of tasks, including COVID-19 detection from chest Xrays, Alzheimer’s classification from MRI scans, and skin lesion analysis, all while significantly reducing dataset size and training requirements. By analyzing the performance, benefits, and trade-offs of both KD and DD, this review offers insights into how these emerging techniques can support the development of more scalable, efficient, and privacy-aware AI systems in healthcare. 2 Knowledge distillation Pour am´eliorer le diagnostic des anomalies sur les IRM c´er´ebrales, plusieurs ´etudes exploitent la distillation de connaissances (Knowledge Distillation, KD) en transf´erant les connaissances d’un mod`ele enseignant complexe vers un mod`ele ´etudiant plus l´eger, afin de maintenir une haute pr´ecision diagnostique tout en r´eduisant la consommation de m´emoire et le temps d’inf´erence. Par exemple, l’approche propos´ee dans [1], utilisant un ensemble de 357 images IRM, a atteint une pr´ecision impressionnante de 98,10 %. FM-LiteLearn, pr´esent´e dans [23], int`egre les images afin d’am´eliorer la repr´esentation des caract´eristiques tumorales ; une strat´egie de distillation multi-enseignants (MT-KD) y est appliqu´ee pour optimiser les performances. ´ Evalu´e sur le jeu de donn´ees BT NAGMN5, le mod`ele propos´e T-ResNet18 a obtenu une am´elioration de 9,4 % de la pr´ecision de classification. Une autre ´etude, pr´esent´ee dans [16], utilise l’auto-distillation (Class Activation Self-Distillation, strat´egie CASD) pour am´eliorer la classification multi-modale du Gliome en affinant l’extraction des caract´eristiques au sein d’un r´eseau `a flux unique. Figure 1shows Knowledge Distillation Framework: Soft Label Supervision from Teacher to Student. Figure 1: Knowledge Distillation Framework: Soft Label Supervision from Teacher to Student Addressing data insufficiency in 3D brain imaging, recent studies [17,26,8] have demonstrated the effectiveness of knowledge distillation (KD) in enhancing model performance with limited data. For instance, in [17], KD improves performance by transferring knowledge from a powerful teacher model to a lightweight student model that combines a convolutional neural network (CNN) for feature extraction with a long short-term memory (LSTM) network to capture inter-slice correlations. This approach achieved an accuracy of 85.96%, representing a 3.83% improvement in Alzheimer’s disease detection using 3D MRI scans. To improve AI transparency in medical image analysis, another study [8] introduces Knowledge Distillation and Feature Map Visualization (KD-FMV), where a DenseNet121 teacher model is trained and transfers its knowledge to a lightweight student model. The method balances hard and soft losses for brain tumor classification: the teacher model achieved 98.77% accuracy, while the best student model reached 97.48% with a lower loss of 0.0944. In Alzheimer’s classification, the teacher model attained 99.38% accuracy, with the best student model achieving 99.46% and a lower loss of 0.0194. On the other hand, the Confidence Regularized Knowledge Distillation (CReg-KD) framework [26] achieved the highest accuracy across multiple architectures: ResNet-18 (∼94%), ResNet-50 (∼93%), DenseNet-121 (∼93%), and InceptionV3 (∼94%), consistently outperforming other distillation methods as the sample size decreased. To address privacy concerns in brain tumor MRI analysis, FedBrain-Distill [7] and FedSPD [24] adopt a federated learning approach combined with knowledge distillation (KD). Results from the Figshare brain tumor dataset (see Figure 2) show that, with two teacher models, FedBrain-Distill achieves over 93% accuracy within just 10 communication rounds, whereas traditional federated learning (FL) methods 49 require 100 rounds to reach similar performance. When using five teachers, the model reaches nearly 94% accuracy, matching state-of-the-art results. FedSPD, on the other hand, employs similarity-preserving knowledge distillation to align feature representations across clients. It outperforms traditional FL methods by 78.41% and personalized FL (PFL) methods by 10.55% in non-IID settings. Moreover, it enhances efficiency by reducing training time by 67.25% and model size by 49.34%. Figure 2: Brain Tumor MRI images from FiGSHARE dataset Other studies, such as [20], utilize multiple teacher models and a lightweight student model incorporating feature aggregation, attention mechanisms, and a custom distillation loss function to improve learning efficiency while maintaining high accuracy. In [20], teacher models are trained on different datasets, while the student learns from both labeled and unlabeled data. Experimental results on the ACDC dataset and various source datasets demonstrate the method’s effectiveness. For instance, on ACDC, the model achieved 89.40% accuracy with only 32 labeled samples, which improved to 95.85% with 64 labeled samples. Likewise, the F1-score increased from 89.42% to 95.74% with more labeled data [19]. Recently, vision transformers (ViTs) have been increasingly applied in conjunction with distillation techniques to transfer both intermediate features and soft labels from ViTs to smaller student models, mitigating the data inefficiency and computational complexity of ViTs in brain tumor MRI classification. LCDEiT (Linear-Complexity Data-Efficient Image Transformer) [6] introduces a custom gated-pooled CNN teacher and an external attention mechanism to improve training efficiency and reduce dependency on large datasets. This strategy achieved high classification performance: 98.11% accuracy and 97.86% F1-score on the Figshare dataset, and 93.69% accuracy and 93.68% F1-score on the BraTS-21 dataset. Hybrid models [5,3] further enhance feature representation while significantly reducing model complexity in the distillation process. The Data-Efficient Knowledge Distillation (HDKD) approach [5] employs a CNN-based teacher model that distills both logitand feature-level knowledge into a hybrid student architecture combining CNN and ViT components with a lightweight convolutional block (MBCSA). This method achieved 92.9% accuracy using only 200 images. Finally, Quantum ViTs (QViTs) [3] leverage KD to pretrain quantum vision transformer models from high-quality teachers, thereby reinforcing the strengths of QViTs. On the OASIS dataset for Alzheimer’s disease classification, QViT 28 (AUC: 0.812, ACC: 0.693) closely rivals ViT 28 (AUC: 0.822, ACC: 0.701) while maintaining quantum efficiency. Furthermore, QViT 224 (AUC: 0.785, ACC: 0.678) outperforms ViT 224 (AUC: 0.603, ACC: 0.400). On Alzheimer’s disease classification tasks, the study [22] employed a Res-Transformer as the teacher model and a ResU-Net as the student model to enhance training stability and optimize skip connections for improved image reconstruction. Through knowledge distillation, this approach achieved an accuracy of 96.9% and a gain of 7.2% in performance. In another study [4], a Residual Temporal Attention Block (RTAB) was introduced to distill temporal dependencies in a dual-stream vision transformer (DS-ViT). The method transfers knowledge from a segmentation model to a classification model by computing residuals between MRI scans over time, guiding the model to focus on subtle cues related to disease progression. This approach achieved 89.9% accuracy and 91.7% recall on the MIRIAD Alzheimer’s dataset. Additionally, the study in [10] explored the feasibility of using vision transformers (ViTs) under lowdata constraints for Alzheimer’s diagnosis. The proposed method reached 79.7% accuracy on the ADNI1 dataset and 82.0% on the ADNI2 dataset, significantly outperforming training without distillation, which achieved only 67.7% and 69.8% accuracy respectively. 3 Dataset distillation Dataset distillation (DD) is a technique that compresses the knowledge of a large dataset into a small set of synthetic images, enabling models to learn effectively while significantly reducing data size and 50 Reference Teacher Model (T) Student Model (S) Distillation Strategy Performance [8] KD-FMV DenseNet121 (27.37 MB) Custom CNN NB params: 37M Feature matching + soft targets 98.77% (T), 97.48% (S) [7] FedBrainDistill 2 or 5 decentralized VGG16 teachers NB params: ∼138M ×(2/5) ConvNet 94,986 params Federated soft distillation 5 teachers: 94.38% (IID)(S) 93.34% (non-IID) (S) 2 teachers: 93.60% (IID)(S) 92.36% (non-IID) (S) [6]LCDEiT Gated pooled CNN (Figshare - 3 classes) NB params: 90795 BraTS-21 (4 classes): 90828 Transformer D1: ∼338,502 D2: ∼33,632 DEiT (Student) + Gated-Pooled CNN (Teacher) + External Attention Figshare Acc 98.11 (T), Acc :94.33 (S) BraTS Acc 93.69 (T), Acc :87.96 (S) LCDEiT Figshare: Acc:98.11 BraTS: Acc :93.69 [23] MT-KD Multiple large models NB params: N.A T-ResNet18 NB params: 11.7M Multi-teachers (ensemble distillation) +9.4% acc improvement [26] CReg-KD ResNet-18 (NB: ∼11.7M) ResNet-50 (NB: ∼25.6M) DenseNet-121 (NB: ∼7.98M) InceptionV3 (NB: ∼23.9M) ResNet-18, ResNet-50, DenseNet-121, InceptionV3 Self-distillation (same model used) Baseline: Acc 87.05 →KD: 92.35±2.10 Baseline: Acc 88.45 →KD: 92.02±2.48 Baseline: Acc 86.01 →KD: 92.36±2.27 Baseline: Acc 90.21 →KD: 94.05±1.20 Table 1: Performance Comparison of Knowledge Distillation Methods on Brain MRI Images preserving privacy (see Figure 4). The survey presented in [11] categorizes DD methods into two main frameworks: the Meta-Learning Framework, which optimizes synthetic data using methods such as Backpropagation Through Time (e.g., DD,LD,GTN ) and Kernel Ridge Regression (e.g., KIP,FRePo); and the Data Matching Framework, which aligns properties between real and synthetic data through Gradient Match (e.g., DC,IDC ), Trajectory Match (e.g., MTT,TESLA), and Distribution Match (e.g., DM,KFS,IDM ). The survey concludes that Data Matching methods, particularly recent factorized approaches such as IDC,RTP, and KFS, generally outperform Meta-Learning methods on complex datasets such as CIFAR-100 and Tiny ImageNet. Alternatively, a more recent survey [27] proposes a different taxonomy based on four dimensions: optimization objective, network update fashion, synthetic data parameterization, and label learning strategy. It classifies DD methods into Performance Matching (e.g., DD,FRePo), Distribution Matching (e.g., DM,IDC ), and Parameter Matching (e.g., MTT,HaBa). This taxonomy highlights key differences in update strategies, label types, and parameterization styles, showing that modern methods such as FRePo and DSA offer superior scalability and accuracy across both simple and complex datasets. Taken together, both surveys suggest that modern, factorized, and optimization-efficient DD methods—particularly those based on Data Matching—are more effective and scalable, marking a clear evolution in the field. In dataset distillation (DD) for natural images, the structure and patterns of the original data are often preserved. Typically, the distilled dataset is initialized using real images or slightly modified versions, allowing the synthetic data to retain key visual features from the original dataset. This helps maintain important textures, shapes, and object structures, enabling models to learn from a smaller subset without significant loss of information. Recent advances in DD have introduced a variety of innovative strategies to improve efficiency, robustness, privacy, and scalability. The Importance-Aware Adaptive Dataset Distillation (IADD) [15] enhances performance by assigning importance-aware weights to network parameters during training, leading to state-of-the-art results on CIFAR-10, CIFAR-100, and 51 Figure 3: Brain MRI Samples of Alzheimer’s Lesions from the OASIS-2 Dataset Ref Teacher Model (T) Student Model (S) Distillation Strategy Dataset Performance [8]DenseNet121 params: 27.37 MB Custom CNN params: 37M Logit matching + spatial feature map analysis 3D MRI ACC: 99.46% (S) ACC: 99.38% (T) [3]TinyViT params: ∼5M Quantum ViT (QVLT) Soft label KD with quantumclassical hybrid learning OASIS ACC:0.965 AUC: 0.983 (T) ACC: 0.656 AUC: 0.763(S) [22] ResTransformer (U-Net based) NB params: ∼40M–60M ResU-Net params: ∼7M–15M Intermediate features + soft labels Private ACC: N.A (T) ACC: 96.9% (S) +7.2% acc [4] 3D Unet-Model (EastSurfer) NB params: ∼24M DS-VLT params: ∼10–14M Dual-stream distillation (segmentation to classification) MIRIAD ACC: N.A (T) ADAPT ACC: 0.903 (baseline) DS-ViT (S) ACC: 0.941 [10]3D ResNet-152 params: ∼256M Lightweight Transformer params: N.A Distillation token with selfattention ADNI1 ADNI2 ADNI1: ACC : 84.63% (T) ACC : 79.7%(S) ADNI2: ACC 82.51% (T) ACC: 82.0%(S) Table 2: Performance Comparison of KD Methods on Alzheimer MRI Images Tiny ImageNet. In the domain of robustness, TrustDD [18] introduces Pseudo-Outlier Exposure (POE) to distill datasets that are more resistant to out-of-distribution (OOD) inputs, achieving top AUROC and AUPR-OUT scores on CIFAR-10. The method proposed in [29] improves training efficiency through early-stage model snapshots and parameter perturbation, enabling up to 20×speedups without compromising accuracy. Privacy concerns are addressed in SFDD [2], which applies a local differential privacy mechanism (LDPO-RLD) in a decentralized setting, protecting gradient updates while improving model performance, with an 8.94% accuracy gain on the GTSRB dataset. Study [30] proposes a diffusion-based patch selection strategy for synthetic data generation, introducing a novel DD method that uses a frozen diffusion model as a teacher to select informative image patches from real data, rather than generating synthetic images. By leveraging the model’s learned data distribution and text-guided semantics, it ranks and clusters patches to build a compact yet effective distilled dataset. A student model trained on this curated set learns efficiently, preserving the original data’s feature distribution and improving semantic alignment compared to prior approaches. This strategy achieved 70.0% top-1 accuracy on ImageNet-1K. Finally, Distributional Dataset Distillation (DDD) is introduced in [21], a novel approach addressing inefficiencies in prototype-based DD methods, particularly the hidden storage costs of explicit label encoding. Rather than distilling into individual samples, DDD represents each class using compact per-class statistical distributions, coupled with a decoder to reconstruct representative data. This formulation allows significantly more memory-efficient distillation. To enhance scalability, the authors propose a federated distillation strategy by splitting the dataset into subsets, distilling them in parallel 52 Figure 4: Dataset Distillation Framework: performance matching via specialized sub-task models, and merging the results. Extensive experiments demonstrate that DDD achieves state-of-the-art performance, including a +6.9% accuracy improvement on ImageNet-1K under tight storage constraints (equivalent to just two images per class), highlighting both its efficiency and scalability. Collectively, these contributions mark significant progress, as illustrated in Table 3, making dataset distillation more accurate, scalable, privacy-preserving, and efficient across diverse domains and challenges. Ref Dataset Classes Size IPC Model(s) Distilled Performance Original Performance [15] CIFAR-10 10 60000 50 ConvNetD3 ACC = 72.6% ACC = 84.8 ±0.1% [15] CIFAR-100 100 60000 50 ConvNetD3 ACC = 49.0% ACC = 56.2 ±0.3% [18]CIFAR-10 10 60000 10 ConvNet AUROC= 78.64% AUROC= 65.79% 100 60000 10 ConvNet AUROC: 82.04% AUROC= 49.94% 10 60000 50 ConvNet ACC = 60.22% ACC = 60.55% AlexNet ACC = 58.36% ACC = 56.23% VGG ACC = 56.29% ACC = 55.02% [18] CIFAR-100 100 60000 10 ResNet ACC = 51.25% ACC = 50.00% [29] ImageNet-10 10 ˜13,000 10 ResNetAP-10 ACC = 74.6% SpeedUp ×4.57, Acc gain ×1.01 [29] ImageNet-100 100 ˜133,000 10 ResNetAP-10 ACC = 48.4% SpeedUp ×4.76, Acc gain ×1.04 [30] ImageNet-1K 1000 ˜1.28 M 50/100 VLT-B to ResNet-18 IPC 50: ACC=65.4 ±0.7% IPC 100:ACC=70.0 ±0.3% NA [21] ImageNet-1K 1000 ˜1.28 M 10 ConvNet ACC=30.5% +6.9% gain over baseline [2]GTSRB 43 39,27 1 ConvNet ACC=32.13% +0.19% over centralized DD (31.94%) 43 39,27 10 ConvNet ACC=65.38% ACC=66.55% Table 3: Performance Comparison of Dataset Distillation Methods on Natural Images (IPC: Image Number Per Class) However, in medical imaging, privacy concerns require a different approach. Instead of initializing distilled data from real medical images, the process begins with completely random patterns, often generated using Gaussian noise or other noise-based strategies. These synthetic samples are then optimized through dataset distillation techniques to replicate the training behavior of real medical images while ensuring that no identifiable structures or sensitive patient information are retained. This strategy enables privacy-preserving model training while maintaining the effectiveness of the distilled datasets for downstream tasks. In the medical imaging domain, several recent DD methods have been proposed to enable secure, efficient, and privacy-preserving model training. For instance, UniCompress [25] applies dataset distillation to the domain of medical image compression, using feature alignment and cross-attention in a knowledge distillation (KD) pipeline to transfer information from a teacher to a lightweight student model, achieving 4–5×faster compression with top-tier PSNR and SSIM scores. The Anonymous Gastric Image Distillation method [12] uses a gradient-based approach that achieves a harmonic mean (HM) of 0.877, outperforming ResNet-18 trained on up to 3,000 real images. Similarly, soft-label dataset distillation (SLDD) [13] also achieves an HM of 0.877, surpassing both traditional hard-label distillation methods and large-scale ResNet-18 models. The study concludes that the minimum number of compressed images required is correlated with the number of model parameters. 53 For cross-hospital sharing of COVID-19 chest X-ray data, another study [14] achieves 82.7% accuracy using only 20 images per class, closely approaching the 88.9% accuracy obtained when training on the full dataset. MedSynth [9] introduces a condensation framework that uses an attention-based generator fine-tuned with a Vision Transformer (ViT) to align synthetic and real data logits, achieving up to 97.11% accuracy and 96.27% AUC on Alzheimer’s and ISIC 2019 datasets. Further advancing medical dataset distillation, a progressive trajectory matching method [28] applies multi-stage alignment of model parameters trained on synthetic versus real data, combined with dynamic overlap mitigation and scheduled retraining. This approach achieves state-of-the-art accuracy on high-resolution datasets, including 66.18% on COVID19-CXR, 65.37% on BREAST-ULS, and 51.19% on SKIN-HAM, using only two images per class, outperforming all previously reported distillation techniques. Ref Dataset Size D.Size / IPC Model Distilled Performance Baseline Performance [25] CT Scans 201 3D patients ∼512× compression ratio ResNet-50 ACC = 0.9811 (Liver) Teacher models (no DD): slower by 4–5× ACC = 0.9812 (Colon) ACC = 0.9758 (Spleen) [12] Gastric X-ray 815 patients 1 ResNet-18 Sensitivity = 0.886 +5% HM Specificity = 0.869 [13] Gastric X-ray 815 patients 1 GoogLeNet HM = 0.882 Cross-model comparison ResNet-18 HM = 0.869 AlexNet HM = 0.836 VGG16 HM = 0.916 [14]COVID-19 Chest X-ray 21 165 20 ConvNet Accuracy = 82.7% 88.9% (full dataset) [9] Alzheimer’s, ISIC 2019 5 121 samples 50 DCGAN ACC = 97.11% ∼20× smaller dataset [28] COVID19-CXR 21 165 2/10 ConvNet IPC 2: ACC = 66.18% ±0.02 90.22% ±0.01 IPC 10: ACC = 69.65% ±0.01 [28]BREAST-ULS 780 2/10 ConvNet IPC 2: ACC = 65.37% ±0.02 74.00% ±0.07 SKIN-HAM 10 015 2/10 ConvNet IPC 10: ACC =68.90% ±0.01 IPC 2: ACC = 51.19% ±0.02 70.17% ±0.02 Table 4: Performance Comparison of DD Methods on Medical Images (IPC: Image Number Per Class) Future research in dataset distillation should focus on advanced techniques like GANs, VAEs, and diffusion models to enhance data fidelity and privacy-preserving data sharing which is crucial in sensitive domains like medical imaging. Advancements in methods like MedSynth and DDD should emphasize scalability, generalization, and robustness. 4 Discussion Knowledge distillation (KD) in brain MRI imaging has emerged as a powerful technique not only for compressing large models into smaller, more efficient ones but also for addressing challenges such as privacy, diversity, and adaptability to specific target tasks. When integrated with approaches like federated learning (e.g., FedBrain-Distill [5]), it enables model training without directly sharing sensitive data, thereby enhancing privacy. FedBrain-Distill focuses on privacy and communication efficiency in federated settings, where multiple decentralized teacher models produce soft labels (via temperaturescaled softmax) for a central lightweight student, resulting in strong generalization performance without exchanging data or model weights (94.38% IID, 93.34% non-IID). In contrast, deep models such as DenseNet121, used in [6], serve as rich teachers to guide a custom CNN student via both soft targets and feature representation matching, achieving high performance (98.77% teacher, 97.48% student) despite the student having more parameters—highlighting computational efficiency and inference speed as more critical than model size. Furthermore, recent studies have expanded the scope of KD by exploring the use of more complex or “heavy” teacher models to optimize performance, efficiency, and model architectures, adapting the methods for specific applications and deployment environments. For instance, the study in [10] replaces heavy teacher models with a lightweight gated CNN and uses attention-guided and self-supervised distillation to train a compact transformer-based LCDEiT student (approximately 338K parameters), achieving competitive accuracy (up to 98.11%) with far lower complexity. Collectively, these studies demonstrate how KD can optimize performance, efficiency, privacy, and architectural design depending on the specific use case. In the context of Alzheimer’s disease datasets, several studies have shown the power of KD in reducing model complexity without compromising—and in some cases, even enhancing—student model 54 performance. For example, the work in [6] uses KD through logit matching and feature map analysis, allowing a custom 5-layer CNN to approach the accuracy of a DenseNet121 teacher (99.38% vs. 99.46%) while requiring nearly ten times fewer operations, demonstrating how lightweight models can achieve comparable results. In [15], knowledge transfer from a Res-Transformer to a lightweight ResU-Net student results in a 7.2% accuracy increase and an estimated 5–10×reduction in computational complexity, emphasizing the benefits of distilling both soft labels and intermediate features. Taking a novel direction, study [27] explores the integration of KD with quantum neural networks, where a TinyViT teacher distills knowledge into a QViT student model. This approach achieves up to 80×compression while maintaining strong classification performance (AUC up to 0.812), highlighting KD’s potential in hybrid quantum-classical learning systems. Similarly, KD can go beyond modelto-model transfer for the same task. In [18], a dual-stream KD strategy is employed to bridge two distinct tasks—segmentation and classification—by distilling structural knowledge from a segmentation model into a DS-ViT classifier. This cross-task distillation leads to notable improvements in classification accuracy (0.899) and recall (0.917), while also reducing model size by approximately 5×, showcasing KD’s ability to enable transfer learning across functionally different but related domains. Another approach, proposed in [29], introduces a distillation token mechanism that transfers knowledge from a large 3D ResNet-152 to a significantly smaller transformer-based student. Despite achieving 9.7×compression, the performance gain is modest ( 0.1%) and computational costs remain high due to the heavy teacher model. These studies illustrate the broad utility of KD—not only as a model compression method, but also as a bridge across architectures, learning paradigms, and deployment constraints—especially within the sensitive and resource-limited field of medical imaging. Inspired by the principles of KD, a related and increasingly powerful technique known as dataset distillation (DD) has emerged. Unlike KD, which transfers knowledge from one model to another, DD transfers knowledge from real data—or a model trained on it—into a much smaller, synthetic dataset. In this paradigm, the teacher is not a model but the original data distribution itself, and the student (often with the same architecture as the teacher) is trained exclusively on the distilled dataset. This method aims to match the performance of models trained on full datasets using only a synthesized subset derived from the original data. Recent studies have further developed this concept by evaluating multiple model architectures during the distillation process, thereby improving the generalizability of the distilled data across tasks and learners. Among its key advantages, DD significantly accelerates training times and reduces storage requirements, making it especially effective for simpler datasets such as CIFAR-10 and GTSRB. For instance, studies show that with just 10 to 50 images per class (IPC), models trained on distilled CIFAR-10 can achieve up to 72.6% accuracy compared to 84.8% on the full dataset, while GTSRB achieves 65.38% using IPC 10 versus 66.55% with full data. This makes DD ideal for edge deployment or federated learning setups with strict data transmission and storage limits. Additionally, DD can deliver notable speedups (e.g., 4.5×on ImageNet subsets) and even occasional performance improvements over traditional distributed training. However, these advantages come with significant limitations. As dataset complexity increases (e.g., CIFAR-100, ImageNet-100, ImageNet-1K), the performance gap between models trained on distilled versus full datasets becomes substantial. For example, on ImageNet-1K with 10 IPC, accuracy drops to 30.5%, although increasing to 100 IPC improves it to 70.0%. These datasets contain high inter-class variability and complex patterns that are difficult to capture with limited synthetic data. Moreover, the success of DD is highly architecture-dependent: while simple networks like AlexNet and VGG retain relatively high AUC and accuracy on distilled data, more complex models like ResNet and ViT are more sensitive to the quality and diversity of synthetic samples. To mitigate this, some methods introduce teacher-student mechanisms during DD, improving results at the cost of greater pipeline complexity. DD has shown great promise in medical imaging, achieving significant data reduction while preserving performance, particularly with ConvNet and ResNet architectures. Studies indicate that even under extremely low IPC settings, models can maintain strong performance on tasks such as CT scan segmentation and chest X-ray classification. However, a major challenge to generalizability remains: distilled data often overfit to the model architecture they were generated for, reducing reusability across other architectures. Furthermore, on more diverse or clinically detailed datasets, performance tends to decline, and important but rare features may be lost. Most research has focused on convolutional architectures, limiting exploration of modern transformer-based models. Additionally, the interpretability and clinical reliability of synthetic data remain open concerns in real-world medical applications. 55 5 Conclusion Knowledge and dataset distillation are emerging as impactful techniques in medical imaging, particularly in hospital settings where data privacy, sharing restrictions, and limited computational resources are key concerns. KD enables the compression of large models into efficient, high-performing versions, making it ideal for decentralized systems, edge devices, and federated learning environments where raw data cannot be shared. DD complements this by creating compact synthetic datasets that capture the essential patterns of full datasets, enabling training without exposing sensitive medical data. These approaches support privacy-preserving and resource-efficient AI deployment but face notable challenges. KD often involves complex training pipelines with teacher-student models, while DD may struggle to retain diagnostic fidelity in high-resolution or fine-grained tasks. Both methods also show performance variability across different architectures, highlighting the need for careful tuning and optimization. Looking forward, integrating generative adversarial networks (GANs) into dataset distillation offers a promising direction. GANs can improve the realism and diversity of synthetic data, helping to close the performance gap between distilled and full datasets. With continued research, KD and DD have strong potential to support scalable, accurate, and privacy-aware AI systems in clinical environments. References [1] R. Anantathanavit, F. H. Raswa, T. Thaipisutikul, and J. C. Wang. Lightweight brain tumor diagnosis via knowledge distillation. In 2024 International Conference on Multimedia Analysis and Pattern Recognition (MAPR), pages 1–6. IEEE, August 2024. [2] M. Arazzi, M. Cihangiroglu, S. Nicolazzo, and A. Nocera. Secure federated data distillation. arXiv preprint, 2025. [3] T. Boucher and E. B. Mazomenos. Distilling knowledge into quantum vision transformers for biomedical image classification. arXiv preprint, 2025. [4] K. Chen, Y. Wang, Y. Zhou, and H. Wang. Ds-vit: Dual-stream vision transformer for cross-task distillation in alzheimer’s early diagnosis. arXiv preprint, 2024. [5] O. S. EL-Assiouti, G. Hamed, D. Khattab, and H. M. Ebied. Hdkd: Hybrid data-efficient knowledge distillation network for medical image classification. Engineering Applications of Artificial Intelligence, 138:109430, 2024. [6] G. J. Ferdous, K. A. Sathi, M. A. Hossain, M. M. Hoque, and M. A. A. Dewan. Lcdeit: A linear complexity data-efficient image transformer for mri brain tumor classification. IEEE Access, 11:20337–20350, 2023. [7] R. J. Gohari, L. Aliahmadipour, and E. Valipour. Fedbrain-distill: Communication-efficient federated brain tumor classification using ensemble knowledge distillation on non-iid data. arXiv preprint, 2024. [8] Y. Jiang, X. Zhao, Y. Wu, and A. Chaddad. A knowledge distillation-based approach to enhance transparency of classifier models. arXiv preprint, 2025. [9] R. Kanagavelu, M. Walia, Y. Wang, H. Fu, Q. Wei, Y. Liu, and R. S. M. Goh. Medsynth: Leveraging generative model for healthcare data sharing. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 654–664, Cham, October 2024. Springer Nature Switzerland. [10] K. Kunanbayev, V. Shen, and D. S. Kim. Training vit with limited data for alzheimer’s disease classification: An empirical study. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 334–343, Cham, October 2024. Springer Nature Switzerland. [11] S. Lei and D. Tao. A comprehensive survey of dataset distillation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(1):17–32, 2023. [12] G. Li, R. Togo, T. Ogawa, and M. Haseyama. Soft-label anonymous gastric x-ray image distillation. In 2020 IEEE International Conference on Image Processing (ICIP), pages 305–309. IEEE, October 2020. 56