Ensemble methods and semi-supervised learning for information fusion: A review and future research directions

Information Fusion 107 (2024) 102310 Available online 16 February 2024 1566-2535/© 2024 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/bync-nd/4.0/). Contents lists available at ScienceDirect Information Fusion journal homepage: www.elsevier.com/locate/inffus Ensemble methods and semi-supervised learning for information fusion: A review and future research directions José Luis Garrido-Labrador, Ana Serrano-Mamolar∗, Jesús Maudes-Raedo, Juan J. Rodríguez, César García-Osorio Universidad de Burgos, Avda. Cantabria s/n, Burgos, 09006, Spain ARTICLE INFO Keywords: Semi-supervised learning Ensemble learning Information fusion Semi-supervised ensemble classification Label scarcity Bibliographic review Research trends Experimental protocols ABSTRACT Advances over the past decade at the intersection of information fusion methods and Semi-Supervised Learning (SSL) are investigated in this paper that grapple with challenges related to limited labelled data. To do so, a bibliographic review of papers published since 2013 is presented, in which ensemble methods are combined with new machine learning algorithms. A total of 128 new proposals using SSL algorithms for ensemble construction are identified and classified. All the methods are categorised by approach, ensemble type, and base classifier. Experimental protocols, pre-processing, dataset usage, unlabelled ratios, and statistical tests are also assessed, underlining the major trends, and some shortcomings of particular studies. It is evident from this literature review that foundational algorithms such as self-training and co-training are influencing current developments, and that innovative ensemble techniques are continuing to emerge. Additionally, valuable guidelines are identified in the review for improving research into intrinsically semi-supervised and unsupervised pre-processing methods, especially for regression tasks. 1. Introduction Information fusion, an important process that involves integrating and assimilating data from various sources and combining different learner perspectives, has become a critical component in a variety of fields such as natural language processing, computer vision, and affective computing [1]. This comprehensive approach yields a better understanding of complex phenomena and facilitates the creation of more accurate and reliable decisions. However, acquiring labelled data to implement supervised learning methods remains a major obstacle in several information fusion applications. The bottleneck persists, due to high costs, time constraints, and quite often the sheer impracticality of labelling extensive datasets, all of which is compounded by the inherent nature of the data, and in many areas a shortage of domain experts. Additionally, recent advances within the field of information fusion have led to increased interest in objective methodologies such as ensemble learning, which amalgamates diverse models to enhance predictive performance, and Semi-Supervised Learning (SSL) that uses both labelled and unlabelled data for training. Ensemble learning augments decision-making by combining multiple models, effectively mitigating the bias of individual models and improving overall accuracy and reliability. In turn, SSL utilises unlabelled data to supplement the scarcity of labelled samples, addressing the limitations of the all-toooften laborious labelling process. These techniques offer promising ∗Corresponding author. E-mail address: [email protected] (A. Serrano-Mamolar). possibilities in information fusion, providing viable solutions to overcome the constraints imposed by the lack of labelled data in various application domains. SSL has emerged as an outstanding paradigm for addressing the challenges of partially labelled data [2]. The semi-supervised approach takes advantage of the abundance of unlabelled data in many contexts, together with a small set of labelled examples, in order to improve learning performance. Furthermore, ensemble learning is used to improve generalisation, which often solves the problems linked to the poor adaptability of single-learned approaches. Thus, ensemble methods, which combine multiple models to make collective predictions, have shown great potential in SSL. Semi-supervised ensembles can exploit inherent structures and relations within the data, by aggregating the outputs of diverse models trained on both labelled and unlabelled data, to achieve enhanced predictive accuracy and robustness; a process that is also known as information fusion in the field of machine learning. A query search on the Scopus database, using ‘‘semi-supervised’’ and ‘‘ensemble’’ as the search terms, for papers published between 2013 and 2023, returned 450 papers, which are addressed in this review. Among those papers, 128 new semi-supervised ensemble methods were identified in 127 studies. The methods were categorised according to the semi-supervised approach (wrapper, pre-processing, intrinsically), https://doi.org/10.1016/j.inffus.2024.102310 Received 29 September 2023; Received in revised form 14 February 2024; Accepted 15 February 2024 Information Fusion 107 (2024) 102310 2 J.L. Garrido-Labrador et al. Fig. 1. VOSViewer keyword co-occurrence clustering view. Each term is represented by a circle, where its diameter and the size of its label represents the frequency of occurrence of the term. The lines in the visualisation indicate that the two keywords are present in the same paper, and the width of the line is proportional to the number of paper that share these keywords. the ensemble type (Bagging, Boosting...) and the base classifier type. Moreover, the experimental protocol, settings, pre-processing methods, number of data sets employed in the experimentation, unlabelled ratios, and statistical tests were all analysed. A keyword co-occurrence map was generated, based on the set of results found in Scopus using VOSViewer software [3]. A thesaurus was also compiled, in order to perform data cleaning: linking up different spellings of the same term, abbreviated keywords with full keywords, and synonyms. A total of 31 keywords were identified that met the criterion of having a minimum frequency of 3 occurrences from the initial pool of 210 keywords. Then, a co-occurrence analysis was performed on these 31 keywords, as shown in Fig. 1. In this map, the 5 clusters that were identified are shown in different colours. Clustering in VOSviewer involves a process of minimising distances between keywords and placing the most closely related keywords in one cluster [4]. The node area and the font size depends on the weight of the keyword: the higher the value, the more frequent the keyword, and therefore the larger the corresponding node and label. The connecting line between nodes represents a shared occurrence of a keyword with another keyword. The keyword co-occurrence strength is represented by the thickness of the connecting line: the thicker the connecting line, the more frequent the co-occurrences between both keywords. In the red-coloured cluster, the most frequently repeated keywords ‘‘ensembles’’ and ‘‘semi-supervised learning’’ are represented, followed by the term ‘‘classification’’, together with machine learning methods, and algorithms such as ‘‘co-training’’, ‘‘adaboost’’, and ‘‘random forest’’. The yellow group includes the term ‘‘high-dimensional data’’ and related applications, such as ‘‘social-networks’’ and ‘‘spammer detection’’. The blue and purple clusters include machine learning terms and some ensemble algorithms, such as ‘‘self-training’’ and ‘‘tri-training’’. Terms related to ‘‘imbalanced learning’’ can be found in the green cluster, together with applications where imbalanced data is a common issue, such as ‘‘intrusion detection’’ and ‘‘anomaly detection’’. Although there are state-of-the-art reviews on this topic, none offer comprehensive analyses of ensembles used in combination with SSL. The main objective of this review is to offer some valuable insight to the scientific community into the use of ensemble methods for SSL. Techniques employed for combining base classifiers, such as Bagging and Voting, are explored. Additionally, an analysis of commonly used base classifiers including Tree and Support Vector Machine (SVM) classifiers are also examined. Likewise, experimental aspects, including the number of datasets and statistical test routines, are employed. A critical perspective is also provided in this comprehensive review that clarifies the use of existing methods and their characteristics. Another objective of this work is to present a comprehensive catalogue of recent semi-supervised ensemble learning methodologies to the scientific community, enabling future research to consider the current state-of-the-art. Consequently, the evaluation of forthcoming literature proposing novel algorithms can be performed against the most up-todate techniques. Moreover, certain inadequacies of current research are outlined in this paper, so that such shortcomings may be remedied in future studies. Although there are already other review articles on SSL methods, certain differentiating characteristics justify the inclusion of this review in the literature on SSL: 1. First, this review is particularly focused on the intersection of SSL and information fusion methods, specifically ensemble methods. By focusing on this intersection, this study goes beyond traditional reviews and provides valuable insight into the synergies and novel approaches that emerge when both methodologies are combined. 2. It also addresses a crucial gap in previous ones, in so far as it provides an updated analysis that incorporates recent advances to SSL, ensuring that readers are aware of the latest methods proposed in the field. 3. A notable feature of this review, unlike previous reviews, is the analysis of the number of datasets, metrics, and methodologies used in each study, whenever new algorithms are compared with existing ones. In most papers, the comparisons between proposed Information Fusion 107 (2024) 102310 3 J.L. Garrido-Labrador et al. and existing methods are not methodologically substantiated which might otherwise add reliability to their claims that the new methods actually represent improvements. 4. Lastly, an analysis of the keywords used in the papers that are under review, the growing interest in the different SSL methods, and the citation graph and network of links between the papers are all features that cannot be found in other reviews. The remainder of this paper is organised as follows. The basic concepts, assumptions, and taxonomy of SSL methods are covered in Section 2. In Section 3, the review of works on ensembles for SSL and their analysis is organised into three temporal stages: past, present, and future. Finally, the conclusions of the review are given in Section 4. 2. Semi-supervised learning As the name implies, semi-supervised learning is a machine learning technique somewhere between supervised learning, which requires having a labelled dataset, and unsupervised learning, which aims to discover patterns and interesting structures in unlabelled datasets. In semi-supervised learning the ultimate goal is the same as in supervised learning: to obtain a predictive model that can assign labels to unlabelled instances. However, in addition to the labelled instances that are available, semi-supervised learning also utilises instances for which their labels are unknown. The information contained in these unlabelled instances can be exploited in various ways. In the simplest approach, the unlabelled instances are used only at the beginning of the learning process to initialise parameters of a supervised algorithm or to identify groups on which the algorithm will act. Another approach involves iteratively and incrementally expanding the set of labelled instances by adding new instances as the models are refined and confidence in the prediction of labels for unlabelled instances increases. Finally, there are methods that directly use both labelled and unlabelled instances in the computation of the loss function. This section presents the taxonomy of all these semi-supervised learning methods, but first, let us consider the characteristics that a dataset should have to ensure that the use of unlabelled instances can benefit the learning process. 2.1. Underlying assumptions It is important to note that semi-supervised learning is not always guaranteed to improve a supervised model. For unlabelled data to help build a better classifier it is important that sufficient unlabelled data is available and that the distribution of the unlabelled data meets some assumptions [5]: •The smoothness assumption: If two instances look similar, they should actually be of the same class. •The low-density assumption: Class decision boundaries should avoid areas of high density and prefer areas where there are few instances. •The manifold assumption: Instances appearing in the same lowdimensional manifold should have the same class. •The cluster assumption: Instances that are clustered together should be assigned the same class. This principle is a generalisation of the other three. 2.2. Taxonomy and SSL methods classification SSL has traditionally been divided into inductive and transductive learning, depending on the primary goal. While the aim of transductive methods is to obtain the labels of unlabelled data points within a dataset, the aim of an inductive method is to find a generalised model that can generate predictions for any object in the input space. Van Engelen and Hoos [2] presented a novel taxonomy for semisupervised methods (see Fig. 4). The first division separates the previously presented inductive and transductive methods. The methods Fig. 2. Iterative process of a wrapper model [6]. 𝐿is the supervised set,𝑈the unsupervised set,𝑈′is a part of 𝑈with pseudo-labels. Fig. 3. General structure of a semi-supervised algorithm based on unsupervised pre-processing. 𝐿is the supervised set, 𝑈the unsupervised set. used in the transductive branch are called graph-based. They have to undergo three phases: the first concerns the way that the graph is built; the second, the way in which weights are attached to the links; and the third, the way that the class of the unlabelled nodes can be inferred. Inductive methods are divided into three branches depending on how they exploit the unlabelled data. Firstly, wrapper methods, which use an iteratively trained supervised method whose training data includes both instances from the unlabelled data and pseudo-labels (i.e., labels predicted from prior iterations of the model). These techniques can be self-training, if a single classifier is used, co-training, if several are simultaneously employed, and boosting, if several are sequentially utilised. The main difference between these approaches stems from the way that pseudo-labels are included in the labelled subset. An example of this process can be found in Fig. 2. In the unsupervised pre-processing methods, the labelled and unlabelled data are separately used, commonly using the unlabelled data for extraction or transformation of the dataset, and to initialise some of the algorithm parameters. In most of those methods, rather than modifying the labelled set by adding new instances and pseudo-labels, unsupervised techniques are used on the unlabelled (or the completed training set). Typical models include those that extract features, those that apply clustering to propagate classes, and those that include pretraining using autoencoders. Fig. 3 presents a general overview of the above-mentioned methods. Additionally, intrinsically semi-supervised methods are characterised by directly exploiting all the assumptions. Deep-learning methods, in which the objective functions of the unlabelled instances are directly considered, constitute most of the intrinsically semi-supervised methods. There are several methods for SSL with deep learning. A recent review can be found in [7]. It is noteworthy that deep learning methods are intentionally omitted from this study. A decision that was based on two primary considerations: first, image processing is primarily targeted in most of the deeplearning approaches, a very distinct domain quite unlike the other methods under study; second, their operational characteristics differ significantly from the other algorithms that are included in the study. Hence, it would be advisable to conduct a dedicated review exclusively Information Fusion 107 (2024) 102310 4 J.L. Garrido-Labrador et al. Table 1 Pros and cons of inductive methods. Source: Ramírez-Sanz et al. [8]. Licensed CC-BY. Method Pros Cons Wrapper methods 1. Easy to implement. 2. Configurable, easy to change base estimator/s. 3. Can be used with almost any supervised method. 1. Prone to add noise. 2. Dependent on supervised methods. Unsupervised pre-processing 1. Can be used with almost any supervised method. 1. Less impact of unlabelled data. Intrinsically semi-supervised 1. Unlabelled data is used on the lower level (objective function or optimization procedure). 2. Usually easy to develop from its supervised version. 1. More complex models ∼Harder to train. 2. Most of them require large amounts of data. Fig. 4. The taxonomy of semi-supervised methods as proposed in [2]. In addition to the branches within the taxonomy, some representative methods are also shown in each sub-category (some in grey because their categorisation is questionable, or because they could be assigned to more than one category). is used to represent the set of unlabelled instances, and 𝜃to represent the parameters of algorithms. for the examination of semi-supervised methods that employ deep learning. There are enormous differences between intrinsically semi-supervised methods when leveraging the same assumptions. Four categories are identified in the literature, including margin maximisers which utilise the low-density assumption, to create dividing lines via clustering and SVM. Manifold methods exploit the same assumption, so as to generate topological variations that help achieve better data separation. Generative models rely on adversarial models, and finally, perturbation-based methods usually incorporate neural networks that alter the acceptance function or the loss function. Ramírez-Sanz et al. [8] underlined the pros and cons of all of these methods, which can be seen in Table 1. 3. Ensembles for semi-supervised learning Ensemble methods have emerged as effective strategies for improving the generalisability and robustness of predictive models by integrating the predictions of multiple base estimators constructed using a given learning algorithm or combination thereof [9]. In the aforementioned paper, for example, an overview of ensemble methods used in semi-supervised environments can be consulted. 3.1. Past In 2013, Triguero et al. [10] published a review of techniques for SSL within wrapper methods. They delved into the state-of-the-art at that time, focusing on wrappers, most of which employ ensemble learning techniques in the well-known co-training approach. A taxonomy for wrapper methods was introduced, encompassing the categories of single/multi view, single/multi learning, and single/multi classifier. The authors described multi-learner as the combination of various techniques to build the base classifiers, whereas multi classifier involved the use of multiple classifiers to create an ensemble. It suggests that there are no multi-learner methods that function as single classifiers, but there are single learning methods that function as multi classifiers. In that study, the performance of the selected methods was exhaustively explored using fifty-five UCI classification datasets and varying the ratio of labelled instances in the range of 10 to 40 percent of the total instances in the training sets. A total of 18 methods were investigated, of which 14 were co-training methods, thus utilising ensemble techniques. Although the exploration was quite in-depth for the analysis of each method’s performance, it was not notably extensive, as only 18 methods proposed since the emergence of semi-supervised methods over the past 15 years were examined. Nevertheless, their focus remains on the most significant algorithms that have made an impact, as is evident in their continued consideration Information Fusion 107 (2024) 102310 5 J.L. Garrido-Labrador et al. in more recent reviews [7,11]. Additionally, comparisons were drawn with other methods introduced in recent years [12,13]. Over the past 10 years, additional reviews on SSL have been published, such as the one by Ning et al. [14], which was specifically focused on Co-Training. The authors provided detailed information on the fundamental stages of such an algorithm: view acquisition, learner differentiation, and label confidence assessment. In terms of view acquisition, they highlighted five methods [14] for introducing diversity into classifiers: generating random subspaces, with the RASCO [15] and the Rel-RASCO [16] algorithms as prominent examples; adopting independent views, as initially proposed by Blum and Michel in the original Co-Training algorithm [17]; ensuring the sufficiency of views, as demonstrated in RSCO [18]; utilising automated partitioning, as found in the CODA algorithm [19]; and the more recent strategy of segmenting views based on knowledge space, an approach that only has one reference specifically oriented towards deep learning: the DeCoTa algorithm [20]. Regarding learner differentiation, the use of different base learners was considered in such methods as Democratic CoTraining [21]. Moreover, the use of different optimisation algorithms was likewise proposed in [22] where genetic algorithms and particle swarm optimisation were used to obtain slightly different SVM; and different ways of setting the parameters of the base learner, for example, in [23] where the Minkowski distance exponent was changed to obtain two different regression models within a co-training algorithm. Additionally, they analysed different ways with which to measure and to maintain learner differences. The final parameter taken into consideration was the evaluation of pseudo-label confidence. The same authors drew a distinction between implicit confidence derived from classifier certainty and explicit confidence using ten-fold cross-validation techniques, among others. Despite such an interesting perspective in their proposal to characterise the different co-training methods, they only considered some of the early methods, neglecting the most recent. A more recent review focused on self-training was authored by Amini et al. in 2022 [24]. Among the methods they considered, only four used ensemble techniques. Much like the review of Triguero et al. [10], whose more in-depth than broad analysis included an empirical study, while only focusing on a few selected methods. 3.2. Present In the last 10 years there have been many new contributions to semi-supervised learning that have not been covered by previous literature reviews, making them outdated. In this section, we review articles published since 2013, focusing on the combination of semi-supervised learning with ensemble construction methods. In what follows, the search and filtering process of the reviewed articles is explained, as well as the identification of subgroups depending on the base method, the use of pre-processing techniques or the size of the data sets. Finally, a study of the experimental validation carried out in the articles is presented. 3.2.1. Search and filtering process Research into semi-supervised ensembles over the past decade is thoroughly explored in this review paper. Its aim is to provide a comprehensive understanding of the current state of research and its evolutionary trajectory. To do so, the following search terms were used to extract a set of papers from the Scopus database: (KEY (ensemble) AND KEY (semi-supervised)) AND PUBYEAR > 2012 Fig. 5. Number of algorithms in each of the semi-supervised types proposed by Van Engelen and Hoos [2], some of which appear in more than one category. This initial search yielded a total of 450 papers, and a preliminary filtering process was carried out to include only those that introduced new methods, while excluding reviews and applications. As a result of this initial filter, a total of 186 new semi-supervised ensemble methods were identified, spanning various categories. Two additional studies that fitted the search parameters were also included, which were referenced in some of the sample papers. Subsequently, to refine the analysis towards these algorithms, the focus was narrowed down to those associated with classification and regression, leaving a total of 128 SSL methods. It required the exclusion of certain branches of SSL, including transductive learning, which lacks predictive capacity, despite functioning with both classifiers and regressors, and active learning, due to its reliance on user intervention. The 127 papers for the review were classified according to the taxonomy of Van Engelen and Hoos [2]. The results showed that 94 of the papers proposed wrapper methods, 16 were on intrinsically semisupervised methods, and 21 on unsupervised pre-processing methods. A certain subjectivity may be noted in this categorisation, in so far as four of the papers fell into more than one category, hence the sum of the papers within each category was not equal to the total number of papers. An UpSet plot [25] with intersections is shown in Fig. 5. The intersections are shown within a matrix, with the matrix rows corresponding to the sets, and the columns corresponding to the intersections between those sets. The size of the sets and the intersections are shown as horizontal and vertical bar charts, respectively. The distribution of the papers over the years and the three main categories are shown in Fig. 6. As expected, most methods were wrapper methods and the large number of papers published between 2018 and 2021 is striking. 3.2.2. Analysis of methods With a specific focus on ensemble methods, this study encompasses a wide range of techniques functioning in accordance with the cotraining and the boosting paradigms, both of which are frameworks embedded within wrapper methods. Another analysis was conducted with regard to the ensemble techniques and the most common base classifiers. These techniques are often combined. The most widely used techniques are weighted and majority voting (both represented as ‘‘Voting’’ in Fig. 7), followed Information Fusion 107 (2024) 102310 6 J.L. Garrido-Labrador et al. Fig. 6. Evolution in the last ten years of the production of algorithm grouped by type and year. Fig. 7. Number of algorithms that use the different types of ensembles or combinations thereof that were described in the sample of papers. Another six combinations were used in only one article. All possible combinations can be seen in the Appendix A with their respective papers. The Set Size shows the global size, including the combinations that are not displayed. by methods which employed Bagging, Forest,1and Boosting. Apart from the minority methods grouped in the category ‘‘Other’’ (which includes the use of the softmax function and Bayesian networks, among others), stacking was the least used method. The UpSet plot in Fig. 7 shows the different combinations of techniques. In 101 of the 127 papers, (80%), the authors preferred to use a single technique instead of a combination. A total of 2 or more techniques were combined in 26 of the 127 papers, the most commonly used combination being Bagging-Voting. The most widely used base classifiers were decision trees, followed by the SVM algorithm. Those results were expected, as the SVM was one of the first algorithms adapted for semi-supervision [26] and decision trees were among the most widely used algorithms for ensemble classification [27,28]. Extensive usage of Artificial Neural Networks was also noted, such as Multilayer Perceptrons, and lazy techniques, such as KNN. To a lesser extent, some techniques were based on Bayes’ theorem and linear methods, the latter being mainly for regression problems. The distribution diagram of these base classifiers can be found in Fig. 8. The combination of learners obtained with different 1For this study, Random Forests, despite being a sub-type of bagging, has been treated as a distinct category with other methods such as Rotation Forest and CoForest as ‘‘Forest’’. methods is a common practice in SSL where it is usually referred to as Co-Learning [10,21]. The type of base estimator determines whether it is a classifier or a regressor. Over the past ten years, algorithms for semi-supervised regression were presented in only 13 articles; the remaining 114 algorithms presented over that same period were for classification. An interesting point for evaluation is whether the algorithms in the papers under review utilised any form of pre-processing. A total of 45 algorithms made use of some type of pre-processing, among which 7 employed two or more methods. Those methods primarily included subspaces and feature extraction, along with a wide variety of techniques such as normalisation, feature selection, manifolds, and noise reduction. The distribution can be observed in Fig. 9. 3.2.3. Evaluating the experimental validation The way that the experimentation was conducted to evaluate the proposed algorithms is another detail for assessment. It includes the number of datasets used, labelling ratios, whether train–test splitting or cross-validation was employed, and the number of executions, as well as whether statistical tests were conducted in comparison with other models, specifying the statistical methods used for such comparisons. In Fig. 10, a histogram is presented in intervals of five, illustrating the number of datasets used for experimentation. The most common Information Fusion 107 (2024) 102310 7 J.L. Garrido-Labrador et al. Fig. 8. Distribution of base learner methods used in the proposed ensembles. There are seventeen other combinations, which can be either Co-Learning, or testing with various techniques, used only in one article. All possible combinations can be seen in Appendix A alongside their respective references. The Set Size shows the global size, including the combinations that are not displayed. Fig. 9. Pre-processing techniques used in some algorithms. There are a further seven combinations, only used in one paper. All possible combinations can be seen in the supplementary material. The Set Size shows the global size, including the combinations that are not displayed. approach is to use less than ten datasets. In relation to the labelling ratio, a fixed number of labels were utilised in twelve algorithms for the entire dataset or a fixed number per class. One labelling ratio was tested in a total of 28 papers. The most common minimum and maximum labelling ranges were both within 0.1−0.15. Train–test splitting was the most typical evaluation after cross-validation techniques, with a ratio of 63 train–test splits to 37 cross-validations. Concerning the number of executions, only one was reported in most papers, with multiple executions described in only 40 of the papers. The mean was 21 executions and the most frequent value was 10 for the papers reporting more than one execution routine. The general trend is unfortunately not to employ any statistical test, so comparisons with other methods are complicated. In Fig. 11, it can be observed that no tests of any type were used in most of the papers, totalling 73 (57%). In decreasing order of usage, the tests were as follows: Friedman test, Student-ttest, Wilcoxon test, Nemenyi test, Holm test, Bonferroni test, and Bayesian test. Additionally, there are some others, such as Iman–Davenport or Finner used in certain articles. It was noteworthy that Bayesian tests, proposed by Benavoli et al. in 2017 [29], were used in only one of the papers found in this study. Demšar also proposed many statistical tests in 2006 [30], so it is all the more remarkable that after so many years, algorithms without statistical tests are still employed in many studies that might otherwise have yielded some statistical verification of the improvements that were proposed in those studies. In cases where statistical tests were utilised, the typical alpha values were 0.01 and 0.05. Lastly, it is interesting to discuss which metrics are most commonly used to assess the performance of the algorithms. Here, a distinction is made between classification and regression, as the metrics differ. The intersection diagrams can be observed in Figs. 12 and 13. The metrics used in classification are quite diverse, with a notable prevalence of accuracy, found in 84 papers and exclusively used in up to 49 papers, often presented as its inverse, the error rate. That metric is followed by the F-Score metrics (primarily F1), precision, recall, the area under the ROC curve, and Cohen’s kappa. Additionally, other metrics that are less commonly used are the geometric mean and specificity. In contrast, the Root Mean Square Error was the most common regression metric, followed by the coefficient of determination and some other variations, such as the Relative Root Mean Square Error, and the Mean Absolute Error. Numerous studies have been conducted, revealing two main types of research. The first type involves general methods that are tested on a large number of datasets. The second type involves methods created to solve specific problems by exploiting a particular dataset. This group includes several noteworthy examples that highlight the proposed solutions to various problems. One new method for detecting adverse effects in drugs is the SSEL-ADE [31]. This recent study proposes a Co-Training and random subspaces-based approach, using a dataset that combines various data sources. Another example worth highlighting is the EnSSL algorithm [32, 33], which has been tested for medical problems such as blood and lung diseases. The algorithm combines several classical semi-supervised techniques, including Tri-Training, Co-Training, and Self-Training. Information Fusion 107 (2024) 102310 8 J.L. Garrido-Labrador et al. Fig. 10. Histogram of articles using different numbers of datasets. Fig. 11. Statistical tests applied to evaluate the performance of the proposed algorithms. There are a further eight combinations only used in one paper. All combinations can be seen in the supplementary material. The Set Size shows the global size, including the combinations that are not displayed. 3.3. Discussion and future guidelines After extensive analysis, certain aspects of current machine learning techniques may now be discussed. The first aspect concerns the prevalence of wrapper methods, specifically those utilising co-training techniques. It prompts the question as to why unsupervised pre-processing and intrinsically semi-supervised techniques are not as well-developed. One explanation is that ensembles, which are closely related to cotraining techniques, are perhaps easier to design and to adapt. In any case, exploring the development of varying types of methods would be an interesting avenue to pursue. Another point for discussion is the type of base estimator to be used. SVMs are far less widely used than trees. On the one hand, it may be expected and, on the other hand, it deserves some discussion. Ensemble methods primarily consist of decision trees [27], so their widespread usage is expected. However, it is surprising how the low-density assumption can be exploited in SVMs, one of the oldest semi-supervised methods, by maximising the distances between the frontier data [5]. Another noteworthy aspect is Co-Learning [10]; given the current trend of combining methods, further exploration of those techniques is a compelling area of research. One observable trend is the significant number of studies that have no experimental evaluation. Although all the papers presented experiments and results, over half included no statistical comparison. Additionally, many provided no implementations, posing challenges for new methods to establish themselves as robust techniques. Conducting statistical tests that show a proposed method performing significantly better than other comparative methods offers ample evidence for other researchers wishing to apply it to similar problems. Furthermore, if these scholarly works included access to the full implementation of the researchers, it might facilitate implementation in production environments and encourage further research. An analysis of the citation graph of the papers and the temporal evolution in Fig. 14 led to further exploration of the topic. It is worth noting that 81 out of the 127 papers lacked references or were not referenced in the other papers that formed the sample for this study. Moreover, it was found that most papers that did reference each other showed minimal cross-referencing. Considering the information presented in the previous paragraph, it becomes clear why statistical assurance and accessible implementations are lacking and why relationships are not established. A researcher encounters difficulties when comparing their method to others with no evidence of significant performance, and the implementation process requires interpretation. It should be reiterated that numerous methods cannot be consolidated in the current state of the art. Furthermore, while reviews prove useful in evaluating method performance, such as the one conducted by Triguero et al. [10], their implementation proves time-consuming for each method. Based on the above discussion, several future directions can be identified. 1. There is significant potential to create methods that utilise semisupervised and unsupervised pre-processing techniques, as well as exploring combinations of those methods. 2. The use of more statistical methods to evaluate newly developed methods is recommended, given their recent growth and underutilisation in this field, and further exploration of the Bayesian tests is suggested [29]. 3. Statistical comparisons of SSL techniques with their supervised counterparts is essential when employing supervised methods as baseline learners. 4. It is also considered essential to provide accessible implementations of each new method that is presented, which can be included in such libraries as Weka [34], LAMDA-SSL [35], and the SSL Library (sslearn) [36], and to follow the API indications Information Fusion 107 (2024) 102310 9 J.L. Garrido-Labrador et al. Fig. 12. Classification metrics to evaluate the performance of the proposed algorithms. There are another nineteen combinations, used only in one article. All combinations can be seen in the supplementary material. The Set Size shows the global size, including the combinations that are not displayed. Fig. 13. Regression metrics to evaluate the performance of the proposed algorithms. The Set Size shows the global size, including the combinations that are not displayed. for more established libraries such as SciKit-Learn [37]. Additionally, for greater accessibility, any such implementations can be included in platforms such as Papers with code.2 Beyond these practical recommendations, there are some desirable objectives that could be considered in future work. Rather than guidelines, these objectives are for anyone wishing to delve deeper into the subject. Firstly, conducting more specific studies on which types of problems align better with particular types of data, especially concerning fields of knowledge (industrial, medical, ecological, etc.), data types (text, images, sounds, tabular, etc.), and their combinations. Secondly, statistical assessments of the performance of various methods can be performed on specific datasets. It would be particularly beneficial to have well-populated libraries of well-implemented 2https://paperswithcode.com. methods and specific semi-supervised datasets, moving beyond the typical ‘‘unlabelling’’ approach. This would have significant implications, requiring the proposal of new metrics to evaluate the performance of algorithms in strictly semi-supervised settings, understood as those in which the real class is impossible to determine without manual labelling. Lastly, methods must be created to ascertain how necessary SSL is for specific datasets and to discern quickly whether utilising supervised methods might be a better option. 4. Conclusions A comprehensive overview of the combined use of SSL and information fusion methods, specifically ensemble methods, has been presented in this review paper, focusing on recent developments since 2013. Several visualisation tools have been used to analyse the relation between the sample of papers for review, such as a citation graph, as well as a keyword co-occurrence map that revealed the main topics and trends within the field. Furthermore, UpSet plots have been used to analyse the frequency of use of the ensemble methods, base classifiers, pre-processing techniques, statistical tests, and performance metrics. A critical evaluation has been presented of past and present research into semi-supervised ensembles, highlighting the strengths and weaknesses of different approaches, as well as the experimental aspects and validation methods. Some of the main findings of this review are as follows: •Semi-supervised ensembles have shown their great potential for improving the learning performance and robustness of predictive models by exploiting the unlabelled data and the diversity of base classifiers. •There was a remarkable increase in papers published on the topic between 2018 and 2020. •Wrapper methods were the most popular and widely studied category of semi-supervised ensembles, especially the co-training and the self-training variants. •Bagging and voting were the most common techniques for combining base classifiers, while decision trees and support vector machines were the most frequently used base learners. •There was a lack of extensive and rigorous experimentation and validation in many studies, as well as a need for more citations and influence of the proposed methods. Information Fusion 107 (2024) 102310 16 J.L. Garrido-Labrador et al. [144] O. Gharroudi, H. Elghazel, A. Aussem, A semi-supervised ensemble approach for multi-label learning, in: 2016 IEEE 16th International Conference on Data Mining Workshops, ICDMW, IEEE, 2016, pp. 1197–1204, http://dx.doi.org/10. 1109/icdmw.2016.0173. [145] Y. Liu, X. Li, Z. Zhang, A new approach in reject inference of using ensemble learning based on global semi-supervised framework, Future Gener. Comput. Syst. 109 (2020) 382–391, http://dx.doi.org/10.1016/j.future.2020.03.047. [146] X. Zhang, F. Jiang, R. Zhang, S. Li, Y. Zhou, Social spammer detection based on semi-supervised learning, in: 2021 IEEE 20th International Conference on Trust, Security and Privacy in Computing and Communications, TrustCom, IEEE, 2021, pp. 849–855, http://dx.doi.org/10.1109/trustcom53373.2021.00120. [147] C. Li, Y. Xie, X. Chen, Semi-supervised ensemble classification method based on near neighbor and its application, Processes 8 (4) (2020) 415, http://dx. doi.org/10.3390/pr8040415. [148] X. Niu, W. Ma, Semi-supervised classifier ensemble model for high-dimensional data, Inform. Sci. 643 (2023) 119203, http://dx.doi.org/10.1016/j.ins.2023. 119203. [149] C. Jian, Y. Ao, Imbalanced fault diagnosis based on semi-supervised ensemble learning, J. Intell. Manuf. 34 (7) (2022) 3143–3158, http://dx.doi.org/10.1007/ s10845-022-01985-2. [150] C. Vo, P. Nguyen, ST_OS: An effective semisupervised learning method for course-level early predictions, IEEE Trans. Learn. Technol. 14 (2) (2021) 238–256, http://dx.doi.org/10.1109/tlt.2021.3072995. [151] V.T.N. Chau, N.H. Phung, An effective semi-supervised learning framework for temporal student classification, in: 2019 6th NAFOSTED Conference on Information and Computer Science, NICS, IEEE, 2019, pp. 363–369, http: //dx.doi.org/10.1109/nics48868.2019.9023795. [152] S. Zhao, J. Li, A semi-supervised self-training method based on density peaks and natural neighbors, J. Ambient Intell. Humaniz. Comput. 12 (2) (2020) 2939–2953, http://dx.doi.org/10.1007/s12652-020-02451-8. [153] Y. Wang, T. Li, Improving semi-supervised co-forest algorithm in evolving data streams, Appl. Intell. 48 (10) (2018) 3248–3262, http://dx.doi.org/10.1007/ s10489-018-1149-7. [154] M.-L. Zhang, Z.-H. Zhou, Exploiting unlabeled data to enhance ensemble diversity, Data Min. Knowl. Discov. 26 (1) (2011) 98–129, http://dx.doi.org/ 10.1007/s10618-011-0243-9. [155] R. Lin, O. Li, Q. Li, Y. Liu, Unknown network protocol classification method based on semi-supervised learning, in: 2015 IEEE International Conference on Computer and Communications, ICCC, IEEE, 2015, pp. 300–308, http://dx.doi. org/10.1109/compcomm.2015.7387586. [156] R. Thomas, J.E. Judith, Voting-based ensemble of unsupervised outlier detectors, in: Lecture Notes in Electrical Engineering, Springer Singapore, 2020, pp. 501–511, http://dx.doi.org/10.1007/978-981-15-3992-3_42. [157] I. Livieris, A. Kanavos, V. Tampakas, P. Pintelas, A weighted voting ensemble self-labeled algorithm for the detection of lung abnormalities from X-Rays, Algorithms 12 (3) (2019) 64, http://dx.doi.org/10.3390/a12030064. [158] Y. Zhao, M.K. Hryniewicki, XGBOD: Improving supervised outlier detection with unsupervised representation learning, in: 2018 International Joint Conference on Neural Networks, IJCNN, IEEE, 2018, pp. 1–8, http://dx.doi.org/10.1109/ ijcnn.2018.8489605. [159] X. Li, W. Ma, Z. Zhou, C. Xu, XSS attack detection model based on semisupervised learning algorithm with weighted neighbor purity, in: Ad-Hoc, Mobile, and Wireless Networks, Springer International Publishing, 2020, pp. 198–213, http://dx.doi.org/10.1007/978-3-030-61746-2_15.