Active learning for fraud detection

Universidade do Minho Escola de Engenharia Departamento de Inform´ atica Miguel Lobo Pinto Leite Active Learning for Fraud Detection November 2020 Universidade do Minho Escola de Engenharia Departamento de Inform´ atica Miguel Lobo Pinto Leite Active Learning for Fraud Detection Master dissertation Integrated Master in Informatics Engineering Dissertation supervised by Paulo Azevedo (DI, Universidade do Minho) November 2020 COPYRIGHT NOTICE This is an academic work that can be used by third parties provided that internationally accepted rules and good practice concerning copyright and related rights are respected. Consequently, this work may be used in accordance with the license Creative Commons Attribution-NonCommercial-NoDerivatives 4.0International (CC BY-NC-ND 4.0) —https://creativecommons.org/licenses/by-nc-nd/4.0/. If one needs permission to make use of the work under conditions not foreseen in the indicated license, the author should be contacted through Reposit ´ oriUM of Universidade do Minho. i ACKNOWLEDGEMENTS I want to express my gratitude for those who were part of or contributed to this work. My supervisor and mentor, Marco Sampaio, who managed our project remarkably, provided me with knowledge and guidance throughout the whole process and performed a pervasive review of my dissertation. My other mentor, Ricardo Barata, who brought exceptional work and ideas to our project and was always available to help me. And our past colleague, Ricardo Pacheco, for his contributions and how he revolutionized the project with high standards for how we organized our work. I learned a lot and received much friendship from them, but also from all the others who are part of Feedzai’s Research department. I must thank the company as a whole, which made it possible for me to work at such an exciting and challenging project. And of course, my supervisor from the University, Professor Paulo Azevedo, who was always committed to keeping up with our work, providing his ideas, and ensuring I deliver a well-written document. This work represents the end of my Masters. Therefore, I would appreciate offering my special thanks to the many who had an impact on my education. I would rather not say all the names, but I must emphasize my parents. They are the main reason for this to be possible by providing me with an education, teaching me to persist through challenges, and always guiding me in my decisions. ii STATEMENT OF INTEGRITY I hereby declare having conducted this academic work with integrity. I confirm that I have not used plagiarism or any form of undue use of information or falsification of results along the process leading to its elaboration. I further declare that I have fully acknowledged the Code of Ethical Conduct of Universidade do Minho. iii Assinado por : MIGUEL LOBO PINTO LEITE Num. de Identificação: BI15184822 Data: 2020.11.13 11:25:40 +0000 RESUMO Um obst ´ aculo comum em v ´ arios dom ´ ınios no processo de prepara c¸˜ ao de um modelo de Machine Learning (ML) ´ e a escassez de labels (i.e., etiquetas dos dados). Em aplica c¸ ˜ oes reais, algures no processo de constru c¸˜ ao de um dataset existe um especialista a fazer anota c¸˜ ao manual de cada inst ˆ ancia dos dados para identificar a respetiva label. Dentro do dom ´ ınio de dete c¸˜ ao de fraude, que ´ e normalmente tratado como um problema de ML supervisionado, a exist ˆ encia de analistas de fraude a reverem todas as transa c¸ ˜ oes que ocorrem representaria um n ´ ıvel de custos em recursos humanos inexequ ´ ıvel. Isto leva a que apenas uma fra c¸˜ ao dos dados possam ser manualmente analisados. O sub-campo de ML conhecido como Active Learning (AL) surgiu em resposta a este problema. Em AL s ˜ ao implementados algoritmos que selecionam de forma eficiente quais as inst ˆ ancias dos dados que devem ser analisadas de forma a otimizarem-se os custos de anota c¸˜ ao dos dados. O objetivo principal deste processo ´ e a cria c¸˜ ao de um modelo de previs ˜ ao eficaz treinado com a menor quantidade de dados poss ´ ıvel. Neste trabalho, apresentamos um estudo detalhado de diversas estrat ´ egias de AL em que realizamos experi ˆ encias com dados de aplica c¸ ˜ oes reais. Focamo-nos principalmente no cen ´ ario em que a anota c¸˜ ao dos dados ´ e iniciada a partir do primeiro dia de gera c¸˜ ao dos mesmos, n ˜ ao tendo ` a partida dados pr ´ evios para a constru c¸˜ ao de perfis dos utilizadores nem quaisquer labels. Apresentamos avalia c¸ ˜ oes de novos algoritmos e configura c¸ ˜ oes de AL, assim como m ´ etodos pr ´ e-existentes, atrav ´ es de m ´ ultiplas experi ˆ encias. Estas experi ˆ encias s ˜ ao realizadas num ambiente em streaming (tal como nos sistemas de produ c¸˜ ao em causa), em que as transac¸ ˜ oes s˜ ao processadas em tempo real. Para al ´ em da escolha do algoritmo de AL existem outros par ˆ ametros a definir na configura c¸˜ ao geral. Realizamos estudos que nos permitem compreender quais os valores mais favor ´ aveis de v ´ arios destes par ˆ ametros, incluindo o impacto da escolha do m ´ etodo de pr´ e-processamento de dados e do modelo de ML usado em avaliac¸˜ ao. A maioria dos algoritmos de AL existentes na literatura exigem um conjunto de dados j ´ a com labels que tenha elementos de todas as classes existentes (e.g., transa c¸ ˜ oes leg ´ ıtimas e fraudulentas). Dado que no dom ´ ınio da dete c¸˜ ao de fraude ´ e comum a ocorr ˆ encia de transa c¸ ˜ oes fraudulentas ser rara, isto pode limitar qu ˜ ao r ´ apido um algoritmo de AL totalmente supervisionado pode come c¸ ar a ser utilizado nas primeiras itera c¸ ˜ oes do processo. Em resposta a este problema n ´ os apresentamos uma framework de AL em tr ˆ es fases que utiliza, num per ´ ıodo interm ´ edio, um algoritmo de AL que recorre ` a estrutura dos dados com labels sem utilizar as mesmas. Isto resulta num aumento da efic´ acia do sistema de AL. iv v Dada a hip ´ otese de que dois algoritmos de AL podem ser combinados de forma a produzir um que seja melhor que as suas partes, tamb ´ em desenvolvemos e estudamos v ´ arios m ´ etodos de combina c¸˜ ao destes algoritmos. Realizamos uma compara c¸˜ ao com uma grande quantidade de combina c¸ ˜ oes que nos levam ` a conclus ˜ ao de que tais combina c¸ ˜ oes n ˜ ao aumentam a efic ´ acia relativamente aos algoritmos individuais numa framework de trˆ es fases. Finalmente, realizamos um conjunto de experi ˆ encias em larga escala que cobrem os diversos casos de uso da dete c¸˜ ao de fraude. Os resultados indicam que AL ´ e uma solu c¸˜ ao adequada para os casos de banking emerchant, principalmente quando utilizados algoritmos de AL baseados em incerteza. Contudo, o nosso estudo n ˜ ao demonstrou resultados positivos para um dataset de banking com ocorr ˆ encias de fraude extremamente raras nem para o dataset de merchant acquirer. Keywords— active learning, data science, fraud detection, machine learning ABSTRACT A problem that arises in many domains when preparing a machine learning (ML) model is label scarcity. In various real world applications, somewhere in the loop of building a dataset, there is a human expert manually annotating each dataset entry with the class label it belongs to. In fraud detection, which is usually addressed as a supervised machine learning problem, having fraud experts carefully reviewing every single transaction is often too expensive, so only a subset of them can be manually annotated. The sub-field of ML known as active learning (AL) has emerged to address this problem. AL implements policies that intelligently choose which instances should be labeled by a human annotator in order to optimize the data labelling costs. The ultimate goal of this procedure is to create a robust predictive model with as little data as possible [Settles (2009)]. In this work, we present a detailed study of various proposed AL strategies by performing experiments with real world data. We focus, primarily, on the scenario where the annotation starts from day-one with no previous data to build historical user profiles and, hence, no labeled data. We present evaluations of several new and already existing types of AL policies and AL configurations through various sets of experiments. The analysis is performed in a streaming setup (as required by the production systems under study) where transactions are processed in real-time. Besides the choice of a policy, there are other parameters that must be chosen in our AL setup. We conduct dedicated studies to assess the most suitable choices for several such parameters. These studies include the understanding of the impact on the choice of the data pre-processing methods and the ML model to use in evaluations. Since most AL policies proposed in the literature require that the pool of labeled instances contains labels from all classes, the extreme class imbalance in the fraud detection domain can limit how fast a fully supervised AL policy can start being used in the first iterations of an AL process. To address this issue, we introduce a three-phase AL framework, which uses an intermediate stage policy that does not resort to the label values but can still exploit the labeled pool. This improves the overall performance of all policies used. Based on the hypothesis that two AL policies can be combined to produce one that outperforms each part, we also develop and study several policy combination methods. We perform a comparison on a large set of combinations that leads us to the conclusion that these do not increase performance when compared to the individual policies in a three-phase setup. Finally, we perform a set of large-scale experiments that cover several business cases for fraud detection. The results support that AL is an appropriate solution for the banking and merchant business cases, especially when using uncertainty sampling as final policy. However, our study did not demonstrate good results for a banking dataset with an extremely small fraud prevalence nor for a merchant acquirer dataset. Keywords— active learning, data science, fraud detection, machine learning vi CONTENTS 1 introduction 1 2 state of the art 5 2.1Sampling approaches 5 2.2Active learning techniques 6 2.2.1Uncertainty Sampling 6 2.2.2Query by committee 6 2.2.3Expected Model Change 6 2.2.4Estimated Error Reduction 7 2.2.5Density-weighted methods 8 2.2.6Discriminative active learning 8 2.2.7Adaptive active learning 8 2.3Active learning for streaming data 9 2.4Stopping Criteria 9 3 problem statement &proposed approaches 11 3.1Fraud business cases 11 3.1.1Financial business case 11 3.1.2Merchant business case 12 3.1.3Acquirer business case 12 3.2The challenges of active learning in fraud detection 12 3.3Problem statement 13 3.4System Architecture 13 3.4.1Pre-processing 13 3.4.2Data manager 16 3.4.3Policy manager 16 3.4.4Labeling manager 17 3.4.5Model train manager 17 3.5Experimental framework 17 3.5.1Simulated data stream 18 3.5.2Simulated labeler 19 3.5.3Evaluation Manager 19 3.5.4Processed test data 19 3.5.5Report generator 19 3.6Active learning policies 19 3.6.1Outlier discriminative active learning (ODAL) 20 3.6.2Query by committee 20 3.6.3Expected model change 22 3.6.4Uncertainty sampling 22 3.6.5Density-weighted methods 22 vii 3 • What is a sufficient budget, in terms of how many data instances we are able to label, to consistently produce efficient ML models? •Which ML models are most adequate for the specificities of AL? •Which data pre-processing methods benefit AL the most? •How can we the address extreme class imbalance in AL? •Which AL policies tend to obtain best results? •Can the combination of AL policies work better than its parts? •How big is the influence of label noise? •How much do results vary across several fraud business cases? The software tools and techniques developed in this dissertation can be more widely applied to other use cases. In particular, they have been used recently in Lorenz et al. (2020) to address the money laundering sub-domain of fraud detection. The technical contributions of this dissertation are as follows: • We propose a novel discriminative AL policy that is computationally efficient for very large datasets. It uses the labeled pool without resorting to the labels, which is especially useful at early iterations of AL for imbalanced problems. This is introduced in Section 3.6.1. • A framework for AL that splits the process into three phases, and that is particularly suited to deal with the problem of extreme class imbalance (section 3.7). •Two measures of similarity to compare AL policies (Section 3.8.1), namely: – The similarity between the feature space data distribution of two data subsets. This is based on comparing the distributions of instances in the data clusters between the two pools. We apply this to compare pairs of labeled pools, each generated by a different AL policy. – The similarity between the predictions of pairs of ML models. We apply this to compare two ML models trained each with a different labeled pool. The two labeled pools are produced by two different AL policy (Section 3.8.1). • Two methods to combine different AL policies (section 3.8.2). One is based on weighted scoring and the other is a pipeline of policies that filter instances. • Techniques for the evaluation of AL configurations, such as our learning curves visualization for the analysis of a single run (Section 4.4.1) and Key Performance Indicators (KPIs) that allow to quantify the performance of AL configurations and carry out large-scale comparisons (Section 4.4.2). • A large-scale study where several AL policies are benchmarked for multiple fraud business cases (banks, merchants and merchant acquirers – Section 5.6). The structure of the dissertation is the following. In Chapter 2, we review related work and state of the art in the field of AL. In Chapter 3, we present the financial fraud domain, including our main challenges and potential approaches. It includes system architecture, experimental setup, and AL 4 policies. Chapter 4includes datasets and evaluation criteria for the different AL configurations. In Chapter 5we discuss experimental results and determine the best performing AL configurations. Finally, in Chapter 6, we summarize the main results and conclusions of our experiments as well as future research avenues. 2 S TAT E O F T H E A RT AL is a sub-field of ML with already a few decades of development and several techniques proposed. Given that our goal is to find or develop new policies which best fit our use cases, we now provide an extensive review of the current state of the art [Settles (2009)]. 2.1 sampling approaches When performing AL, it is important to define the data querying procedure, i.e., how the data is selected before an AL method determines which data instances to send to the oracle/labeler. There are several proposed approaches in the literature [Settles (2009)]: • Pool-based sampling — an unlabeled dataset is available and the policy must iteratively select the samples that it finds most relevant to query. • Stream-based selective sampling — the data is incoming through a stream and the policy must decide for each entry, individually, if it should be queried or not. • Query synthesis — instead of selecting samples of the data, the policy generates synthetic samples that represent relevant clusters of the data. The querying method must be chosen considering the use-case/context of the system and also the performance level that it provides. In a fraud detection scenario, it is common that large amounts of transactions stream into the system every second. Even though it is a streaming scenario, the transactions are coming at a such a high rate that no team of analysts could ever review them all. Therefore a pool of unlabeled data will inevitably be formed. For this reason, we prefer the pool-based sampling option, since it is naturally adapted to this data collection context. Query synthesis is not very suitable when the oracle is a human [Lang and Baum (1992)] (which is our target scenario), since it can generate non-interpretable data (in fraud detection analysts often look at a complex graph of transactions to find out which ones are fraudulent). In a pool-based approach there is a specific parameter that must be decided: the batch size, i.e. the number of transactions sent to the oracle on each iteration. In theory, a smaller batch size should improve AL because then the policy is updated more frequently. For example, if the batch size is 100, the last instance will be selected by the system without information on the labels of the other 99. On the other hand, if the batch size is 10, at most only the labels of the previous 9instances will be unknown. However, the smaller the batch size, the higher the computational cost. This can be significant if a heavy AL policy is used or if a ML model is trained after each iteration. Therefore, a careful choice must be made to achieve a good trade off between policy performance and computational performance. 5 2.2. Active learning techniques 6 2.2 active learning techniques In this section we review the AL techniques most frequently discussed in the literature. 2.2.1Uncertainty Sampling Uncertainty sampling is a classic, and one of the simplest, AL techniques. In this method, one starts by training a ML model using the labeled pool instances. Then, the model is used to score the unlabeled pool and, based on those scores, we select the instances for which the model is most uncertain about the label. In our scenario, a binary classification problem, the selected transactions are those for which the model predicts scores closer to 0.5. For binary classification, this is equivalent to selecting transactions with a maximum entropy score (if the score is interpreted as a probability). On each iteration, the model is updated and, as a consequence, its decision boundary is refined. Therefore, in this approach, it is essential to use an effective probabilistic model or a ML model with decision boundaries clearly defined (such as support vector machines) [Lewis and Catlett (1994)]. The main advantage of this method is that it is extremely light, given that it only re-trains the model on each iteration with a labeled pool that should be relatively small. However it incurs into the risk of only refining the decision boundary, which could result in the method ending up trapped near a sub-optimal solution due to lack of exploration [Huang et al. (2014)]. Even though it is the simplest method, uncertainty sampling has demonstrated to usually achieve the best results [Yang and Loog (2016)]. 2.2.2Query by committee Query by committee is a simple, but slightly heavier method, whose goal is to explore regions of the data that are more ambiguous and difficult for various models. This method uses an ensemble of machine learning models, selected by the user, which are again trained on the already labeled data and then provide scores on the unlabeled data. Subsequently, a measure of model disagreement is computed for each instance (usually the entropy of the prediction outputs). The instances on which the committee of models disagree the most are the ones selected for querying. This is called the principle of maximum disagreement [Seung et al. (1992)]. Even though this method appears to be interesting, it has the major disadvantage that there is no “ideal” committee, which means that the choice of committee members is extremely arbitrary. 2.2.3Expected Model Change Based on the principle that a useful instance is one that will impact the model, the expected model change method aims to identify the unlabeled instances that will most impact the parameters of the model if their labels are revealed. To achieve this, first a gradient-based classifier is trained on the labeled data pool. Then, for each unlabeled instance, the contribution of the instance to the gradient of the loss function is computed for each label assignment. Finally a weighted sum of the L2norm of the two possible gradients is computed, which corresponds to the expected gradient norm (see 2.2. Active learning techniques 7 equation in section 3.3of Settles (2009)). Instances with the highest expected gradient norm are the ones that should be queried [Settles (2009)]. This technique is quite simple, light and displays good results in most cases [Yang and Loog (2016)]. 2.2.4Estimated Error Reduction Estimated error reduction is a technique that attempts to estimate which of the unlabeled instances would result in a greater reduction of the error produced when making predictions with the model. Provided that in an AL context there is not enough labeled data to obtain the actual model error, it is estimated through the level of confidence on its predictions of the unlabeled pool (see equation 1in section 2of Roy and McCallum (2001)). For each unlabeled instance, this method fits an updated model for each of its possible label assignments (in the case of binary classification, 0or 1). These models are then used to estimate the new error after each of the possible label assignments. Finally, an estimated error reduction value is computed for the instance by combining the errors of each of the possible label assignments [Roy and McCallum (2001)]. This approach is demonstrated in algorithm 1. The whole procedure is repeated for all unlabeled instances. In summary, in order to select the instances to query on each iteration the following steps are preformed: 1. a ML model is fitted on the labeled pool; 2. for each instance in the unlabeled pool: a) a copy of the model is retrained with the instance, assuming its label is negative. b) another copy of the model is retrained with the instance, assuming its label is positive. c) the estimated error reduction is computed for each copy of the model and they are combined to obtain the total estimated error reduction for the instance. 3. the instances with the greatest total estimated error reductions are the most relevant ones. Algorithm 1Estimated Error Reduction. 1:model =MODEL.train(labeled data) 2:error reductions = [] 3:for instance ∈unlabeled data do 4:model 0 = copy(model).retrain(instance, 0) 5:model 1 = copy(model).retrain(instance, 1) 6:estimated error 0 = model 0.estimated error(unlabeled \instance) 7:estimated error 1 = model 1.estimated error(unlabeled \instance) 8:error reductions.append(estimated error 0 + estimated error 1) 9:return max index(error reductions) This method is expected to work well with the disadvantage that it is computationally extremely heavy. This makes it impractical for our application, due to the amount of transactions we have to handle. Nevertheless, in Yang and Loog (2016) we can observe that uncertainty sampling and expected model change sometimes outperform it. In its original form this algorithm might not satisfy computational efficiency constraints. Other versions attempt to estimate the error reduction in a lighter way. For example, the method in Fu et al. (2018) uses a sampling technique to select a small 2.2. Active learning techniques 8 number of instances to estimate the estimated error reduction (as an alternative to using all the unlabeled data). 2.2.5Density-weighted methods Uncertainty sampling assumes that the most relevant instances are those closer to the decision boundary. However, instances can simultaneously be outliers and close to the decision boundary. Such instances may not be of interest in AL, since a data point that is similar to no others may bring less useful information for a ML model. Density-weighted methods aim to select instances that are most representative and that cover all of the data distribution. This can be achieved through, e.g., density computation methods or clustering algorithms [Settles (2009)]. Therefore, even though these approaches can provide good data representations, they are usually computationally heavy. A tree-based clustering approach, proposed in Wang et al. (2017), seems to obtain a good performance, however it comes with a relatively high computational cost. 2.2.6Discriminative active learning In Gissin and Shalev-Shwartz (2019) a method is proposed that aims to create a labeled pool that is indistinguishable, in a distributional sense, from the unlabeled pool. The hypothesis is that a small but representative labeled pool will provide a good ML model. This method is implemented by i) training a binary classification model that attempts to detect to which pool an instance belongs, and ii) then it queries the unlabeled pool instances for which the auxiliary model is more confident they do not belong to the labeled pool. The rationale is that these instances are not well represented in the labeled pool so they will bring new information if labeled. This can be interpreted as an outlier detection algorithm which scores the instances of the unlabeled pool that are the greatest outliers relative to the labeled pool. Therefore, we can extend this type of approach by using any outlier detection algorithm that allows scoring instances not used in training, such as Liu et al. (2008), Breunig et al. (2000), Rousseeuw and Driessen (1999)orSch ¨ olkopf et al. (1999). In this dissertation, we will propose experiments with this new approach, named Outlier detection Discriminative Active Learning (ODAL), further explained in section 3.6.1. 2.2.7Adaptive active learning In active learning, different methods typically have different advantages or shortcomings, and follow different principles. Therefore, the question arises whether it is useful to make combinations of different policies. Following this reasoning, adaptive active learning was proposed in Li and Guo (2013), with the goal of combining uncertainty sampling with a density-weighted method to reduce the risk of lack of representativeness in uncertainty sampling. Furthermore, the policy also uses the estimated error reduction method, but only on a small selection of instances with high ranking (as given by the first combination of uncertainty sampling and density-weighted methods) to avoid its high computational costs. This method was named adaptive because it adaptively selects the weights assigned to each of the two policies when scoring instances with the uncertainty sampling 2.3. Active learning for streaming data 9 and density-weighted policies. The performance of this policy has previously shown good results – see also Yang and Loog (2016). 2.3 active learning for streaming data Some studies have appeared in the literature discussing AL methods in a streaming data scenario [ ˇ Zliobait ˙ e et al. (2011); Zhu et al. (2010b); Zhang et al. (2019); Carcillo et al. (2018)]. Notably, in Carcillo et al. (2018), several AL methods were investigated from a perspective of data visualization, for a credit card fraud dataset. In that simulation study, a budget of instances were selected with AL, to be labeled by analysts, once a day. For some of their methods, a budget was also reserved for semi-supervised labeling using a model trained on the labeled data. In contrast we consider scenarios where several small batches of instances are processed during the day to exploit the collected labels more frequently. Furthermore, we will present a detailed analysis of AL curves in the fraud domain, to give a more complete understanding of the effectiveness of AL for fraud, which is not available in Carcillo et al. (2018). Finally, most of the other studies we found in our literature review are either: • focused on applying AL to address concept drift, which is a challenge we will not address at this stage. •not focused on highly imbalanced problems. •not focused on designing an AL method that deals with the cold start. For these reasons, we did not find any of the techniques presented in these studies to be of use within our own. 2.4 stopping criteria An important question when performing AL is when to stop querying data. When performing AL in a real life scenario, usually there is no test dataset. Furthermore, in a streaming scenario, creating one in parallel doubles the costs of AL and might not provide a good representation of the real data distribution of the future streaming data. In Settles (2009), it is stated that the motivations for choosing specific stopping criteria in AL are usually grounded by economic or other external factors and that its objective can be perceived in two ways: •The costs of obtaining more labels is higher than the gains that those labels might bring; • The active learner performance has achieved a plateau and more queries will not bring any significant improvements. Vlachos (2008) proposed an approach that uses the uncertainty of the scores of predictions on the unlabeled data pool to evaluate the confidence of the model. Above a predetermined confidence threshold the querying process can then be stopped. This method has the caveat of requiring that the machine learning model is a strongly probabilistic one. One problem that might occur with this method is, as explained in the section 2.2.1, that the exploration focuses solely on exploiting the decision boundary, resulting in lack of exploration. The model might become too confident on the instances that it has already seen while not ever selecting 2.4. Stopping Criteria 10 important ”unseen” instances. A possible solution to this, would be to combine this stopping criterion with one that would analyze the existence of outliers in the unlabeled data pool compared to the labeled pool, in an identical fashion to the method explained in the section 2.2.6. Several metrics for representing uncertainty as a stopping criteria were proposed in Zhu et al. (2010a), such as: • Maximum uncertainty is the uncertainty score for the instance the model is most uncertain about. •Overall uncertainty is the mean of the uncertainty scores for all instances. • Selected accuracy, first queries the top-minstances on which the model is most uncertain. Once they are labeled, it computes the accuracy, i.e., how many were correctly predicted by the model; The thresholds for these metrics to trigger the stopping must be user-predefined. However, Zhu et al. (2010a) also proposes a threshold update strategy to avoid the issue of defining an ideal threshold for each AL application. In short, every time the stopping criterion is met with the current threshold this strategy verifies if the ML model predictions on the unlabeled pool have changed since the previous threshold update. If that is the case then the threshold is updated to a more restrictive one. Otherwise, if the predictions have not changed, the process stops. This dissertation, however, is mostly focused on evaluating AL policies through simulations using historical data. Therefore the stopping criteria will not be a concern in our experiments (we can run a simulation experiment for any streaming period and observe the results by evaluating on a test set). This brief review is included for completeness and as reference for future work. 3 PROBLEM STATEMENT & PROPOSED APPROACHES In this chapter, we present some the different business use cases where AL can be applied within fraud detection as well as particular challenges for AL in this domain. We also go through the several proposed methods that we will experiment towards the goal of finding an ideal AL system. Specifically, we present: •The overall architecture of our deployable AL system. •The overall architecture of the experimental framework used to benchmark several configurations of the system. •The various AL policies to be evaluated in our experiments. •How we decompose the AL process into phases. • More complex methods to combine different AL policies, as well as methods to assess which policies could be worth combining. 3.1 fraud business cases The central goal of this dissertation is to investigate a set of AL policies that can be used to build a predictive model in a situation where labels are scarce in the fraud domain. However, within this field there are several diverse sub-domains, which are victims of very different fraud patterns. Since there is never a single ideal AL policy for all cases [Yang and Loog (2016)], our goal is to investigate an effective AL policy for each business case. There are three main business cases for transaction data, each with several specific fraud types, which we now discuss in the following sub-sections. 3.1.1Financial business case The financial business case involves the provision of products and services to banks and other financial institutions. As such, many of the risk patterns found in this context are related with movement of funds that can occur using multiple channels such as ATM, transfers, credit and debit cards. Additionally, in the financial context it is important not only to look at each transaction by itself, but also to analyze the profile of each individual/entity, considering account behavior. One particular and quite big problem within this business case is money laundering. This is the act (or attempted act) to conceal or disguise the proceeds of illegal activities so that they appear to 11 3.2. The challenges of active learning in fraud detection 12 come from legitimate sources of activities. The tools and techniques developed in our work have been used to tackle that problem in Lorenz et al. (2020). 3.1.2Merchant business case A merchant is any business engaged in the sale of goods or services. But, only merchants that accept bankcards as a form of payment are pertinent to our explanation. So with that disclaimer, a merchant is any business that maintains a merchant account that enables them to accept credit or debit cards as payment from customers (cardholders) for goods or services provided. 3.1.3Acquirer business case An acquiring bank is a registered member of the card associations (Visa and MasterCard). An acquiring bank is often referred to as a merchant bank because they contract with merchants to create and maintain accounts that allow the business to accept credit and debit cards. Acquiring banks provide merchants with equipment and software to accept cards, promotional materials, customer service and other necessary aspects involved in card acceptance. The acquiring bank also deposits funds from credit card sales into a merchant’s account. 3.2 the challenges of active learning in fraud detection AL may be extremely useful in the fraud domain because obtaining a high quality labeled dataset is expensive – it requires hiring fraud analysts to spend a substantial amount of time labeling transactions. However, there are some challenges specific to this domain, namely: • The data is streaming. The existence of a growing pool of unlabeled data is not standard in AL contexts. In fact, many state of the art methods require a fixed sized unlabeled data pool, which makes them unsuitable for our problem (e.g., most density-weighted methods are not directly applicable, as explained in section 2.2.5). • In the domain of financial transactions, it is common to have to deal with a magnitude of thousands of data instances incoming per minute. • Given that analyzing transactions in order to detect fraud can be a complex and difficult task, a lot of data noise at the label level is to be expected. Regarding the computational challenges, all solutions proposed in this work must comply with the following pre-requisites: • Must be computationally efficient enough to work with large amounts of data. The complexity of fitting the policy must not grow with the unlabeled pool size. • Cannot rely on a fixed unlabeled pool to be efficient, i.e., some methods (such as some density weighted method, in section 2.2.5), are slow because they must fit on the available unlabeled pool (which is a growing problem if the unlabeled pool is continuously expanding). Also, it is expected that label noise makes it harder to build efficient ML models with small quantities of data. We will address this by using highly regularized ML models (section 4.3). 3.6. Active learning policies 19 3.5.2Simulated labeler Since we simulate the flow of data, the historical data source used already contains labels. Hence, we implement a simulated labeler that, when it receives a request to review a batch of transactions, accesses the labels already stored in the historical data source. In order to also simulate somewhat accurately the time it takes for an average proficient analyst to review transactions, our simulated labeler reviews 1000 transactions per day. Hence it signals the system that 86.4 seconds have passed for each labeled transaction (i.e., 24 hours ×60 minutes ×60 seconds /1000). 3.5.3Evaluation Manager The evaluation manager, whose purpose is to measure the quality of the labeled pool at each iteration, is used every time a new batch is labeled. Once the labeled pool is incremented with this batch, a ML model is trained from scratch with the whole pool. Then, this model scores the test data and the configured performance metrics are computed (e.g., area under the ROC curve, recall at some false-positive rate, etc.) and stored. 3.5.4Processed test data The pre-processing pipeline might contain operations that must be applied to the raw data collected in the data stream (e.g., dimensionality reduction). Thus, the output of the pipeline can be highly dependent on the data on which it was fitted (even then it might be stochastic depending on the pre-processing algorithm used). For this reason, data that is used by the system must go through the exact same pre-processing pipeline. For a consistent simulation, this pipeline is trained on the first unlabeled pool before the AL process is started (e.g., data collected in an initial waiting period of one day). This pipeline, besides being used to process the rest of the streaming data, is also stored in a persistent way. Hence, whenever an evaluation is performed, the configured test set is processed using this pipeline, thereby preparing the data to be scored by the model. 3.5.5Report generator In order to be able to visualize results of a single experiment-evaluation pair, we introduce the report generator. This accesses results, such as the metrics stored in the evaluation, to automatically produce a report containing learning curves with their variance bands as well as Key Performance Indicator (KPI) metrics (to be detailed in section 4.4). 3.6 active learning policies In this work we implemented several policies mentioned in chapter 2as well as new variations and new methods, which we now describe in further detail. 3.6. Active learning policies 20 3.6.1Outlier discriminative active learning (ODAL) Given that AL is about creating a pool of unlabeled instances with minimal size, in this section we propose a policy that follows a principle similar to the discriminative AL method [Gissin and Shalev-Shwartz (2019)]. As explained in section 2.2.6, at each iteration that method finds which instances of the unlabeled pool are less well represented in the labeled pool. Our policy, however, achieves this goal using a different approach. On every iteration, we train an outlier detection algorithm on the labeled pool and then use it to score every instance of the unlabeled pool. Our assumption is that the greatest outliers of the unlabeled pool relatively to the labeled one are the transactions that are worse represented in the labeled pool. Hence, these transactions should be reviewed and added with their labels. As outlier detection methods, we experiment with Isolation Forests [Liu et al. (2008)] and Elliptic Envelope [Rousseeuw and Driessen (1999)] 1 . ODAL has the great advantage that it does not require labels to be used. This is helpful in the fraud domain, where the classes are extremely imbalanced and sometimes finding a first positive case in AL might take a lot of time. It can also be considered a computationally light solution, given that the outlier detection algorithm is trained solely on the labeled pool, which should be relatively small (and orders of magnitude smaller than the unlabeled pool). 3.6.2Query by committee As explained in 2.2.2, query-by-committee (QBC) is a method where several models, belonging to a committee, are trained on the labeled pool. Then the level of disagreement between models, on the predictions of the unlabeled pool, is used to define which transactions are most relevant to query. The models we used in our committee were as follows (for all unspecified hyper-parameters the default values from the scikit-learn library were used, since they typically provide finer control that only needs to be changed for very specific problems): •A random forest with 100 trees and max depth of 3. •A logistic regression. •Gaussian Naive Bayes. • Gradient boosting ensemble, with early stopping at 20 iterations without improvement (this implementation requires at least two instances with label from each class for the model to be trained). Since each model outputs scores with a different distribution of values (they are of a different nature), we implemented our disagreement measure using the differences between rankings in the classification instead of the score entropy proposed originally in Seung et al. (1992). Further implementation details are presented in algorithm 3. 1 All the ML models used in this work are implementations from the scikit-learn package [Pedregosa et al. (2011)]. For further details on the models and their defaults, see https://scikit-learn.org/0.22/user guide.html. 3.6. Active learning policies 21 Algorithm 3Query by committee disagreement measure. Require: predm is the list of prediction arrays from every model in the committee. #models is the number of models in the committee and #pred is the number of predictions per model, which is the number of instances in the unlabeled pool. 1:ranksm←[] . Initializes a list for the arrays with rankings of each prediction per model. 2:for predictions ∈predmdo 3:prediction ranks ←rank(predictions).Computes the rank of each prediction. 4:ranksm.append(prediction ranks) 5:di f f erencescombination ←[] .Initializes a list for the differences of each combination of two models. 6:i1←0 7:while i1<#models do 8:i2←i1+1 9:while i2<#models do 10:rank di f f erences ←absolute(ranksi1−ranksi2).Computes the absolute differences array between the ranks. 11:di f f erencescombination.append(rank di f f erences) 12:i2+ = 1 13:i1+ = 1 14:di f f erence means ←avg(array=di f f erencescombination, axis=0).Computes the average differences for each instance. 15:scores ←normalize(di f f erence means).Performs normalization so that scores are between 0and 1. 16:return scores 3.6. Active learning policies 22 3.6.3Expected model change Expected model change, as explained in section 2.2.3, is a policy that requires the usage of a gradientbased ML model. It is used to compute the expected change that each transaction of the unlabeled pool would cause if the current model was trained with it. The higher the expected model change due to an instance, the more relevant its label is. In the experiments we use the scikit-learn logistic regression model with default hyper-parameters. 3.6.4Uncertainty sampling As discussed in section 2.2.1, uncertainty sampling queries instances for which the model produces scores that represent a higher uncertainty level. We implemented this method using three different approaches: • model score distance to 0.5— the instances whose score is closer to 0.5are the ones considered most uncertain. This method has the disadvantage of the used threshold (0.5) being independent of the model. In some scenarios, this value might not represent uncertainty (e.g., when training a random forest with a dataset that has very few positive labels, we observed, in some of our experiments, that the model only outputs scores below this threshold) • distance to fraud percentile — the fraud rate of the labeled pool is computed and then that value is used to compute the percentile of the predictions on the unlabeled pool, which is then used as uncertainty threshold. • epistemic uncertainty — as defined in Shaker and H ¨ ullermeier (2020), uncertainty can be separated in two main categories: aleatoric and epistemic. Aleatoric uncertainty is intrinsic to the data generating process and can never be removed. Epistemic uncertainty can be reduced by collecting more data. We use the implementation proposed in Ascens ˜ ao et al., which instead of total uncertainty only uses the epistemic uncertainty as measure of relevance for labeling. The uncertainty sampling methods are also highly influenced by the choice of the model used. In our case we chose to use the same model and hyper-parameters that we are also using for evaluation. 3.6.5Density-weighted methods Density weighted methods attempt to create a labeled pool that covers the whole feature-space of the data by identifying clusters and querying from them all (section 2.2.5). We implement a variant of these methods. Briefly, for every batch sent for review, this method works in the following steps: 1. An unsupervised ML model identifies the clusters of the whole data (labeled and unlabeled pools). We experimented with the K-means and DBSCAN algorithms [Xu and Wunsch (2005)] (implementations from Pedregosa et al. (2011)), with several different sets of hyper-parameters. 2. The relative frequencies of instances from the labeled pool in each cluster are computed. 3. Instances in the unlabeled pool from clusters that have lower relative frequency values are the ones chosen for review. 3.7. Active learning phases 23 As speculated in section 3.2, this method turns out to be unfeasible to use in our experiments. The growth of the unlabeled pool requires the unsupervised model to be fitted on every iteration, which results in a very high computational cost. 3.7 active learning phases The experiments we will present begin with an empty labeled pool, as explained in section 1. Therefore, at the start of our AL process, data queries must be performed without resorting to the labeled pool. Hence, we implement our framework to support multiple phases, particularly an initial phase of unsupervised querying, as discussed below. 3.7.1Two-phase active learning In the first batch of experiments to be presented, we use only two phases, each with a policy of the following types: 1.Cold policy — a method that does not use the labeled pool and selects instances at random or using an outlier detection algorithm trained on the unlabeled pool. 2.Hot policy — a method that starts being used only after the cold policy found enough positive instances (this constraint is necessary because most of the models used by these policies, except for the discriminative ones, require positive instances in the labeled pool). 3.7.2Three-phase active learning In the second round of experiments, we apply a three-phase AL strategy, introduced to solve a problem detected in the two-phase AL experiments. Specifically, given the very low prevalence of the positive class in the fraud domain, we will see that some AL training runs only switch to the hot policy very close to the end of the established query budget. This behavior is not desirable because then some runs may only use the random policy from start to end without exploiting an AL policy that uses the labeled pool. Therefore, we introduce an AL process composed of three phases, each with a different policy, as follows: 1.Cold policy — One single batch of randomly queried data2. 2.Warm-up policy — A method that uses both pools but not the label values (e.g., a discriminative policy). 3.Hot policy — A supervised AL policy triggered when at least one fraudulent instance is found, except in the case of the query-by-committee policy, for which at least two are needed (see discussion in section 3.6.2). 2 We will see in the two-phase experiments that, in some cases, the outlier detection methods are superior to random sampling. However, that behavior is not stable across different time folds. Since the cold policy is only used for a single batch, and the random policy is computationally more efficient and unbiased, we opted to run three-phase experiments only with a random cold policy for the first batch. 3.8. Policy combination 24 3.8 policy combination Different AL policies follow different sampling principles, each with its advantages and disadvantages. Therefore, one can hypothesize that different policies might query data from different regions in the feature space, resulting in prediction models that, while with similar overall performance, might classify specific instances differently. This hypothesis motivates the use of policies that combine sets of different policies. A combination may yield better performance than single models since it can potentially cover more regions of the feature space. 3.8.1Policy divergence diagnostic In order to decide if we should combine AL policies and, if so, which ones should be combined, we introduce a method to analyze the divergence between two policies. For this assessment a metric to compute the divergence/distance between two data distributions (i.e., the labeled pools) must be defined. In this section we will introduce two such metrics, the cluster frequency distance and the model disagreement distance. Since most AL policies are stochastic and each random seed likely results in different labeled pools, when comparing two policies we run several experiments for each of them with Ndifferent seeds, to be able to evaluate the policy’s variance. We can also exploit this data to estimate the distribution of divergence values between any two policies. Therefore, given a divergence metric, all combinations of pairs of labeled pools between the two policies are compared, which results in a total of (N 2) divergences to be computed (e.g., for 35 seeds we obtain 595 pairs). Then, we assess the degree of divergence between the two policies by comparing the distribution of divergences between their labeled pools, with the distributions of self-divergences between each policy with itself. For example, given two policies with a distribution of divergences that appears large, if the distributions of self-divergences are equally large, then the difference cannot be considered significant. Cluster frequency distance As stated before, it is possible that different policies are fetching data from different regions of the feature space (see illustration in figure 4). Therefore we introduce a metric to compute a measure of similarity (or divergence) between two data pools that works in the following steps: 1. It uses an unsupervised ML algorithm to cluster the whole labeled data (this is the union of all Nlabeled pools in both policies). 2. For each pair of labeled pools to compare, it computes the frequencies of instances that each pool has in the clusters, i.e., a histogram of cluster label frequencies for each pool. 3. It computes the Jensen-Shannon divergence (JSD) [Lin (1991)] between the two histograms, which serves as the measure of divergence between the two data pools. This process is summarized in the algorithm 4. 3.8. Policy combination 25 feature 1 feature 2 feature 1 feature 2 Policy 1 Policy 2 Figure 4: Illustration of the final labeled pools from two different active learning policies which cover different regions of the feature space. Algorithm 4Distance between two data pools. Require: L1 and L2 are the final labeled pools for policy 1and 2, respectively; CLUSTER is a clustering algorithm already trained on all of the data. 1:labels1←cluster(L1).Gets the cluster labels for each pool 2:labels2←cluster(L2) 3:bins1←histogram(labels1) 4:bins2←histogram(labels2) 5:distance ← JSD( bins1,bins2 ) . Computes the Jensen-Shannon Divergence (JSD) between the two histograms’ bin frequencies 6:return pooldistance 3.8. Policy combination 26 Model disagreement distance The metric introduced in the previous section is only sensitive to the distribution of the features regardless of the label. Another different approach is to define a metric that compares model predictions instead. Two different AL policies likely generate different labeled pools. When training two equal ML models, each with one of the two pools, they might each produce better predictions on different instances of the data. In order to compute a measure of distance between two ML models we use the ratio between the number of instances on which the models disagree and the total number of scored instance, which we refer to as ratio of prediction disagreement. This is detailed in algorithm 5. Algorithm 5Distance between two ML models based on prediction disagreement. Require: m1 and m2 are predictive models trained on the final labeled pools for policy 1 and 2, respectively; Xtest and ytest are the test set features and labels, respectively. 1:pred1←m1(Xtest).Obtains the models’ predictions on the test set 2:pred2←m2(Xtest) 3:#disagreements ←0 4:#accurates ←0 5:i←0 6:while i≤#ytest do .Iterates through ytest,pred1and pred2 7:if predi 1=yi test &predi 2=yi test then 8:#accurates =#accurates +1 9:if predi 16=predi 2then 10:#disagreements =#disagreements +1 11:i=i+1 12:distance ←#disagreements/(#disagreements +#accurates) 13:return distance 3.8.2Policy combination methods In this section, we introduce three methods for combining AL policies: one based on weighted-scoring, another one on alternating policies, and the last one on a pipeline of policies that filter the instances. The second one is a sub-type of the first one, as discussed further. Weighted ranking combiner An approach to policy combination based on uncertainty and information density through weighted ranking, has been proposed in Li and Guo (2013). This is applicable to any weighted combination of policies through the formula fw(x) = p1(x)×w+p2(x)×(1−w) where x is the instance being scored for ranking, p1 and p2 are the policies to be combined and w∈[0, 1] is the weight for p1 . The problem of selecting the unknown value of w is addressed by 3.8. Policy combination 27 starting with a predefined set of representative values W={w1,w2, ..., wn} , where the values wi are all the possible values of wto adaptively select in each query (e.g., W={0, 0.25, 0.5, 0.75, 1}). In Li and Guo (2013), the combined policy queries through the following procedure: 1. Finds the top ranked instance for each value of W, by building the set (arg max x∈U fw(x)∀w∈W) 2. Applies the estimated error reduction method, explained in section 2.2.4, on the top ranked instance for each weight wito determine the most interesting one. Even though this method was designed for a batch size of 1, it can be easily modified to find the top-Ninstances for each value of W, with Nconstrained such that batch size =N×n. In this modified setup we note that the estimated error reduction step might be unnecessary and heavy operation, so we implement a simplified version without it. Then Nand nare then adjusted for the equality to hold so that the batch size is met. Alternating combiner Two policies may be good candidates for complementing each other but score instances in a mutually exclusive way, i.e., data instances that one of the policies scores as highly relevant, the other one scores as not relevant at all. In this case, the intermediate weights of the weighted ranking combiner (e.g., w=0.5 ), would not combine these policies adequately. This is because with intermediate weight values, instances that obtain a high relevance score with one policy and a low one with the other one will always obtain only an intermediate final relevance score. For that reason, we introduce a policy that combines two policies by simply alternating between them. This is easily achieved by using the weighted combined learner with W={0, 1}. Cascade combiner The last strategy we consider for policy combination is to start by pre-selecting Ninstances with policy 1and then filtering the batch size best instances out of those Nwith policy 2. This procedure is defined in algorithm 6. 3.8. Policy combination 28 Algorithm 6Cascade combiner. Require: XU is the unlabeled data pool; p1 and p2 are policies 1and 2, respectively; N is the number of instances for policy 1to select; batch size is the number of instances to send to the analyst. Ensure: N>batch size 1:Xranked ←p1(XU).Ranks XUby the interest of p1 2:Xbest ←Xranked[:N].Filters the best Ninstances 3:Xranked ←p2(Xbest).Ranks Xbest by the interest of p2 4:Xf inal ←Xranked[:batch size].Filters the best batch size instances 5:return Xf inal 4.4. Performance metrics 35 –number of trees: {200, 1000} –maximum depth of the trees: {3, 5} •Support vector machine: –Regularization parameter C:{1, 10} –Kernel: {Radial basis function} •Multi-layer perceptron: –Number of hidden layers: {1} –Number of neurons in hidden layer: {50} –Regularization parameter α:{0.01, 0.5} •Gradient boosting ensemble with decision trees •Gaussian Naive Bayes The proposed models and hyper-parameters cover various levels of regularization levels, which is specially relevant due to the challenge of training with such small quantities of data. For each model with varying hyper-parameters, the level of regularization increases when: •Random forest — the number of trees increases or the maximum depth decreases. •Support vector machine — the value of Cdecreases. •Multi-layer perceptron — the value of αincreases. Even though this is not a very extensive experimentation, there is some diversity in the models used and their sets of hyper-parameters. Results are presented in section 5.3. 4.4 performance metrics We now define the performance metrics that will be used to observe an measure the quality of a single AL configuration, as well metrics to allow the comparison between various large-scale sets of experiments across time folds and datasets. 4.4.1Learning curves The performance of a single AL run is tracked by the performance of the model as the labeled pool grows (explained in section 3.5). In figure 10 we present a visualization of the model performance (vertical axis) as a function of the size of the labeled pool (horizontal axis). The represented performance values are the target metric 2 (e.g., recall at some value of false positive rate), obtained by training the model at each iteration and making predictions on the test set. We can observe that this run’s performance keeps increasing until the labeled pool has around 4000 transactions, where it starts to plateau. However, given the stochastic nature of most AL policies, we cannot draw conclusions from a single run. For that reason, we run all experiments in our study with 35 different random seeds, to 2In this dissertation we do not specify the target metrics being used due to confidentiality issues. 4.4. Performance metrics 36 0 1000 2000 3000 4000 5000 6000 Labeled pool size 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Target metric Figure 10: Visualization of the performance throughout a single AL run. This example was taken from a random policy. Figure 11: Visualization of the variance of the performances throughout 35 AL runs with different random seeds. This example was taken from the same experiment as in figure 10 obtain a measure of the variance of the learning curves. 3 This allows us to compare the stability of different policies. In figure 11 we present an example of a visualization of the aggregation over the 35 runs. For each number of instances in the labeled pool, several percentiles of the performance were computed (using the 35 values) and the bands between the intervals of such percentiles were plotted. This allows us to clearly observe the median (50th percentile), the worst and best case scenarios (0th and 100th percentiles) and the positions of some intermediate percentiles. 3 The choice of 35 seeds provides a reasonable trade off between having a good chance of collecting at least one point in the extreme tails of the distribution of values (for each point of the learning curve) and keeping the number of repetitions small enough to run the experiments in a practical amount of time. In particular, for 35 runs there is an ∼84% probability to observe a point in the 5% upper or lower tail of the distribution - see, e.g., equation 3in Pinto et al. (2019) 4.4. Performance metrics 37 0 1000 2000 3000 4000 5000 6000 Labeled pool size 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Target metric Figure 12: Visual representation of the KPI based on the area below the 50 th percentile of the learning curves. The larger this KPI, the better the policy because it will have achieved the plateau earlier and its value will be higher. The horizontal dashed line represents the full train data baseline value. 4.4.2Key performance indicators (KPIs) Due to the large number of experiments in this study, it is not feasible to analyze and compare all the different results by inspection of the corresponding visualizations. To be able to summarize the performance of a single experiment and to be able to compare various experiments, we introduce two KPIs that allow us to sort experiments: • Area under the 50 th percentile — This metric is represented in figure 12. It is the area below the 50 th percentile line of the plot in figure 11 (the horizontal dashed line is the full train data baseline value, which will be explained further in this section). It gives us a number that represents the median performance of the experiment. The higher this value is, the better the configuration. • Area between the 10 th and 90 th percentiles — This metric is represented in figure 13. It is the area between the 10 th and 90 th percentile lines of the plot in figure 11. Since it represents the difference between the almost worst and best scenarios, it gives us a number to represent the variance in the performance of the runs. The lower this value is, the lower is the variance of experiment, which is good. The direct comparison of area under the 50 th percentile values between experiments provides a good indication of which performs better. However, when observing these values individually, we cannot tell how good they are, e.g., it could be the case that all experiments had bad results simply because the dataset is hard on the given fold. A more meaningful metric is obtained by normalizing these metrics by the corresponding value for a model simply trained with the full two weeks of train data instead of using AL. We call this the full train data baseline. This is conjectured to be the best possible performance one can obtain on the test fold. In figure 12 this is displayed as a horizontal dashed line. By dividing the area under the 50 th by the area of the rectangle below the full train data baseline value, we obtain a ratio of the AL performance by its optimal performance. This normalization also makes it easier to compare KPIs across different test folds. 4.4. Performance metrics 38 0 1000 2000 3000 4000 5000 6000 Labeled pool size 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Target metric Figure 13: Visual representation of the KPI based on the area between the 10 th and 90 th percentiles of the performances. The smaller it is the most stable the policy is because its learning curve will have a lower variance. 5 EXPERIMENTAL RESULTS In section 3, the set of approaches used to find the strongest AL configurations were described. The current Chapter describes the experiments performed towards that goal and the corresponding results. The experiments were designed to try to answer the following questions: 1. Without any of the improvements we propose in previous chapters, how do the proposed AL policies work and which perform best? 2. Which of the proposed data pre-processing methods (section 3.4.1) is more effective for AL? 3. Which ML model and hyper-parameters (section 4.3) are more suitable for training with the small amounts of data in AL? 4. Is our proposal of a three-phase framework for AL (section 3.7) beneficial? 5. Can a combination of policies, using our proposed methods (section 3.8), be better than its isolated parts? 6. Having narrowed down the best configurations by answering the previous questions, how do the AL policies behave across the several fraud business cases (section3.1)? In all the experiments we will present, we only provide results with a fixed batch size of 100. We repeated several of the various experiments, at an early stage, with smaller batch sizes (10 and 50) and did not find differences in performance that were significant enough to justify the heavier computational cost of using smaller batch sizes (which for a batch size of 10 would increase the run time by 10 times). In this Chapter, we start by presenting a set of preliminary experiments on a single dataset, to understand the landscape of AL policies that we have proposed to study (using a simple two-phase setup where the AL policy to study is preceded by a cold policy initialization). Then, we present two dedicated studies to investigate the effect of varying two parameters of the experiments that are not directly related to AL, namely the data pre-processing methods (Section 5.2) and the ML model (Section 5.3). This provides validation for our earlier choices. After these preliminary studies, we move on to assess the usefulness of our main proposals on the same dataset, namely: the three-phase AL framework (Section 5.4), the policy divergence diagnostics, and the policy combination methods (Section 5.5). Finally, after narrowing down the sequences of policies to a set of representative cases to investigate, we conduct a set of large-scale experiments on various datasets to verify the best performing policies in each of the fraud business cases we proposed to study (Section 5.6). 39 5.1. Preliminary experiments 40 Fold 1 Fold 2 AVG rank all AVG VAR all Cold policy Hot policy Normalized Area Rank Normalized Area Rank Isolation forest outlier detection Uncertainty sampling 0.81 2 0.83 1 1.5 0.22 Isolation forest outlier detection Isolation forest ODAL 0.78 4 0.74 2 3.0 0.23 Elliptic envelope outlier detection Uncertainty sampling 0.81 1 0.63 8 4.5 0.29 Isolation forest outlier detection Expected model change 0.81 3 0.69 6 4.5 0.20 Isolation forest outlier detection Elliptic envelope ODAL 0.76 6 0.74 3 4.5 0.22 Isolation forest outlier detection Query by committee 0.75 8 0.73 4 6.0 0.27 Isolation forest outlier detection None 0.68 11 0.70 5 8.0 0.27 Elliptic envelope outlier detection Isolation forest ODAL 0.75 7 0.58 12 9.5 0.27 Random Uncertainty sampling 0.64 12 0.64 7 9.5 0.44 Elliptic envelope outlier detection Expected model change 0.77 5 0.56 15 10.0 0.21 Elliptic envelope outlier detection Elliptic envelope ODAL 0.72 9 0.56 14 11.5 0.28 Random Isolation forest ODAL 0.63 14 0.60 11 12.5 0.40 Random Expected model change 0.63 15 0.61 10 12.5 0.45 Random None 0.59 16 0.61 9 12.5 0.40 Random Query by committee 0.63 13 0.56 13 13.0 0.40 Elliptic envelope outlier detection Query by committee 0.71 10 0.51 17 13.5 0.29 Random Elliptic envelope ODAL 0.58 17 0.53 16 16.5 0.39 Elliptic envelope outlier detection None 0.47 18 0.49 18 18.0 0.39 Table 2: Results of the preliminary experiments. 5.1 preliminary experiments Before investigating in detail the impact of specific components of the AL system, we present a set of preliminary experiments to understand the basic behavior of the various policies we proposed to study. In order to keep this exploration simple and to be able to explore many combinations, we only use two time folds of only one (banking) dataset. To isolate as much as possible the behavior of each policy, these experiments were only performed in two phases: a cold policy (an AL policy that does not require a labeled pool, which we can never remove in our problem setup) and a hot policy. This simplification also allows us to test further combinations, in particular more than one policy implementation of a given type. 1 The switch between policies happens when the third fraudulent instance is found in the labeled pool. This is because we want to fit the policies with several positive cases while allowing to perform the switch at an early stage. For the data preprocessing (section 3.4.1), we use the option where the autoML tool generates many features and then the number of features is reduced with PCA. We chose this option because it takes into account correlations between features in a multi-variate way (i.e., several features at a time). Regarding the ML model we use a random forest with 200 trees and a maximum depth of 3nodes, because we expect it to provide a good level of regularization for the small amounts of data used in AL, while letting us run experiments in a manageable amount of time. Since these are preliminary experiments, we delay the validation of these choices to the next sections. The results are presented in table 2. In this table, the rows are candidate policies (i.e., the cold/hot policy combinations) and the columns are time folds. The values in the ”Normalized Area” columns 1 These experiments were conducted at an earlier stage of the project that was prior to the experiments leading to the 3phases strategy explained in section 3.7– presented later in this section. 5.2. Data pre-processing 41 are the area under the 50 th percentile KPI normalized by the area below the full train data baseline (introduced in section 4.4.2) . The full train data baseline is the performance of the model when trained with all of the transactions in the first two weeks of the time fold (while the AL runs are accessing only the first week). Here we introduce this normalization to facilitate interpreting the policy performance relative to the full train data baseline. This also allows clearer comparisons between different time folds (which may have very different full train data baselines due to concept drift). For each time fold an extra column is added with the ranks of the candidates, according to the normalized area for the fold. We also compute an average of ranks over the two folds that is used to sort the table, so that the candidate policies are ordered by performance. Finally, we introduce a column (”AVG VAR ALL”) with the average variance over the folds (i.e., area between the 10 th and 90th percentiles KPI). The variances were also normalized by the same full train data baseline area. Regarding the results in table 2, we observe the following: • Using an outlier detection algorithm as cold policy is almost always better than using a random policy, especially the isolation forest. • Within experiments with the same cold policy, the variances are mostly similar. This means that it is the cold policy, and not the hot policy, that is having most of the impact on the variance. • The random cold policy introduces much more variance in the experiments than the outlier detection ones. • For each cold policy, using uncertainty sampling is always the best hot policy and isolation forest ODAL is the second best. 5.2 data pre-processing The data pre-processing methods we choose to apply in our experiments will affect not only the instances that the AL policies select, but also the performance the ML models obtained. Therefore, it is desirable to validate if the previously used pre-processing method is good and if it should be used in the remaining experiments. In section 3.4.1several pre-processing methods were presented. We proceed by presenting a set of methods combinations: • Execution of an autoML pipeline containing profiles with 6different time windows, followed by PCA dimensionality reduction to decrease to approximately 90 features. • Execution of the same autoML pipeline as above, but then applying a pair-wise correlation filter in order to produce approximately 90 features. • Execution of an autoML run with fewer profiles (4time windows only and removal of some grouping entities according to domain knowledge, to remove features that are most likely redundant). This component is expected to affect all policies in a similar way (it is not an AL component). Therefore, this simple study should suffice to select a reasonable preprocessing method. A more detailed study of the effect of the pre-processing hyper-parameters is beyond the scope of this study and will be left for future work. For simplicity, these experiments were only performed with a reduced set of policies and datasets. Only two time folds of data were used from a banking dataset. 5.2. Data pre-processing 42 Uncertainty sampling Elliptic envelope ODAL Random Query by committee AVG rank all AVG VAR all Fold 1 Fold 2 Fold 1 Fold 2 Fold 1 Fold 2 Fold 1 Fold 2 Feature engineering time windows Feature selection Area Rank Area Rank Area Rank Area Rank Area Rank Area Rank Area Rank Area Rank 7 days, 1 day, 12 hours, 1 hour, 30 mins, 5 mins PCA (90 features) 0.21 1 0.27 1 0.19 1 0.22 2 0.18 1 0.25 1 0.19 1 0.25 1 1.13 0.15 7 days, 1 day, 1 hour, 5 mins Domain knowledge (201 features) 0.15 3 0.26 2 0.12 3 0.22 1 0.15 2 0.23 2 0.13 2 0.24 2 2.13 0.18 7 days, 1 day, 12 hours, 1 hour, 30 mins, 5 mins Pair-wise correlation selection (90 features) 0.17 2 0.25 3 0.13 2 0.19 3 0.13 3 0.22 3 0.13 3 0.20 3 2.75 0.18 Table 3: Results of the experiments with different pre-processing. Regarding the AL policies, we narrowed them down to: uncertainty sampling, elliptic envelope ODAL, a random policy and query by committee 2 . And again, for simplicity, we used a random forest with 200 trees and a maximum depth of 3nodes. The results are presented in the table 3. In this table, the rows correspond to the candidate methods (i.e., the data pre-processing pipelines) and the columns correspond to metrics for specific fold/AL policy combinations. The values in the ”Area” columns are the area under the 50 th percentile KPI (introduced in section 4.4.2). For each fold/AL group an extra column is added with the ranks of the candidate methods, according to the area under the 50 th percentile KPI. We also compute an average of ranks (used to sort the rows of the table) so that the candidates are ordered by average rank of the performance metric. We use the average ranks and not the average KPIs, for this and all further experiments, because our main goal is to understand which AL policies most commonly outrank the others. By averaging the KPIs this measure would be too susceptible to outlier KPIs and to changes in the overall scale of the performance metric due to drift across time folds. Finally, we also introduce a column (”AVG VAR ALL”) with the average variance (i.e., average along each row of the area between the 10th and 90th percentiles KPI) for each fold/AL policy combination). Regarding table 3, we observe the following: •All candidates exhibit similar low variance. • The data pre-processing pipeline with 6time windows and pair-wise correlation selection is almost always the worst performer. • The method with PCA feature selection, which we used for the preliminary experiments, is almost always the best among the candidates. Since explainability is not a priority at such an early stage of the project, we choose to keep using the best performing method for the remaining experiments, despite containing PCA feature selection, which produces less interpretable features. 2 At this early stage exploration we used AL with only with 2phases (the 2versus 3phase setup is explained in section 3.7). 5.3. Model comparisons 43 Uncertainty sampling Elliptic envelope ODAL Random Query by committee AVG rank all AVG VAR all Fold 1 Fold 2 Fold 1 Fold 2 Fold 1 Fold 2 Fold 1 Fold 2 Model Parameters Area Rank Area Rank Area Rank Area Rank Area Rank Area Rank Area Rank Area Rank Random Forest #trees=1000 max depth=3 0.23 1 0.29 1 0.19 1 0.23 1 0.19 2 0.26 1 0.21 1 0.26 1 1.13 0.13 Random Forest #trees=1000 max depth=5 0.21 2 0.28 2 0.18 2 0.22 2 0.19 1 0.25 2 0.21 2 0.24 2 1.88 0.14 Random Forest #trees=200 max depth=3 0.20 3 0.25 3 0.18 3 0.21 3 0.18 3 0.24 4 0.19 3 0.22 3 3.13 0.14 SVM C=1 kernel=rbf 0.05 6 0.05 8 0.14 4 0.14 4 0.11 4 0.24 3 0.14 4 0.10 5 4.75 0.23 Multilayer perceptron hidden layer size=50 alpha=0.5 max #iterations=2000 0.07 5 0.11 4 0.09 6 0.11 5 0.07 6 0.13 5 0.08 7 0.11 4 5.25 0.08 Gradient boosting classifier None 0.08 4 0.09 5 0.07 7 0.07 7 0.07 5 0.10 6 0.09 6 0.08 6 5.75 0.13 SVM C=10 kernel=rbf 0.04 8 0.06 6 0.10 5 0.09 6 0.04 8 0.08 7 0.10 5 0.07 7 6.50 0.23 Multilayer perceptron hidden layer size=50 alpha=0.01 max #iterations=2000 0.05 7 0.05 7 0.05 9 0.05 9 0.05 7 0.07 8 0.05 9 0.06 8 8.00 0.05 Gaussian Naïve Bayes None 0.03 9 0.05 9 0.06 8 0.06 8 0.04 9 0.05 9 0.05 8 0.06 9 8.63 0.05 Table 4: Comparison of the AL performance for several models. 5.3 model comparisons The ML model used for our experiments must be suited to an AL setup. Particularly, this model is to be fitted on small amounts of data, therefore the level of regularization is especially relevant. In section 4.3, we proposed a set of models to study this dependence on the choice of the evaluation model. The results of the corresponding experiments are present in table 4. In this table, the rows are evaluation models (and the respective hyper-parameters) and the columns contain groups with specific time fold/AL policy combinations. Similarly to the results from the previous section, the values in the ”Area” columns are areas under the 50 th percentile KPI (introduced in section 4.4.2). For each group we include a column with the ranking of the models, according to the area under the 50 th percentile. We also compute the average of all ranks (in the ”AVG rank all” column) which is used to sort the rows of the table, so that the models are ordered from higher to lower performance. Finally, we also introduce a column (”AVG VAR ALL”) with the average variance over all groups (i.e., the average area between the 10 th and 90 th percentiles KPI). Similarly to the data pre-processing pipeline experiments (section 5.2), this provides a study of the effect of changing the ML model and its hyper-parameters to determine the best choice for the remaining experiments, so the time folds and AL policies used are the same. In the results table we can observe that: 5.4. AL with 3phases 44 • The more regularization the better — models of a fixed type with hyper-parameters values that provide greater regularization always perform better (the relevance of this topic was also discussed in section 4.3). This is observable in the performance gain of the following models: – Random forests, with either a larger number of trees (for fixed maximum tree depth) or a smaller maximum depth (with fixed, and large, number of trees). – SVM, with a smaller value of the inverse regularization parameter C(smaller values equate to larger regularization). – The multilayer perceptron with a larger value of α , which also results in stronger regularization. • Some models have considerably lower performance and lower variance, such as the Gaussian Na ¨ ıve Bayes and the multilayer perceptron. In those cases, the lower variance is justified by the fact that the performance never reaches larger values. • The random forest algorithm, despite the hyper-parameters used, is generally the best performer. • Within the random forests, the top 3models, the differences in the hyper-parameters have a moderate impact on the absolute value of the areas. We consider that the moderate positive impact of the random forest’s hyper-parameters on AL performance, especially using a much larger number of trees, does not outweigh the additional computational complexity, which would result in much longer training time. For this reason, in the final experiments we will continue using a random forest with 200 trees and a maximum depth of 3 nodes. 5.4 al with 3 phases As explained in section 3.7, most of the hot policies we are using resort to the labels in the labeled pool. This prevents them from starting sooner if there are only labels from one class in the labeled pool (i.e., there must be both negative and positive class instances). This can be a problem for datasets with a high class imbalance, like in the fraud domain, because there will be a delay in the switch from cold to hot policy. Motivated by this observation, we introduce an intermediate stage with a policy that resorts to the labeled pool but not to its labels, so that the collected labels can start being exploited earlier. We call this the warm-up policy stage. In order to evaluate the benefits of this approach, we conduct a simple set of experiments. The only policies that can be used as warm-up policies are of the ODAL type (see section 3.6.1), since they do not use label values. We discard the usage of the elliptic envelope ODAL, because in section 5.1we observed that it under-performed the isolation forest ODAL. In section 5.1, we also concluded that using random as a cold policy instead of outlier detection increases variance and decreases median performance. However, in a three-stage strategy the cold policy is only used once on the first (typically small) batch when the performance is still low regardless of the cold policy method. Since the random policy is computationally much lighter and it is unbiased3, we argue that it is more suitable for the first batch. 3 Despite the good results for the outlier detection warm-up for this banking dataset, it is not clear if the same type of bias will help in other datasets. 5.6. Large scale experiments 51 Fold 1 Fold 2 AVG Rank AVG VAR Hot policy Policy 1 Policy 2 Norm. Area Rank Norm. Area Rank Uncertainty sampling None None 0.68 3 0.76 1 2.0 0.05 Weighted ranking combiner Expected model change Uncertainty sampling 0.69 1 0.73 5 3.0 0.05 Cascade combiner Uncertainty sampling Isolation forest ODAL 0.66 6 0.75 2 4.0 0.04 Alternating combiner Isolation forest ODAL Uncertainty sampling 0.65 7 0.74 4 5.5 0.05 Cascade combiner Query by committee Uncertainty sampling 0.66 5 0.73 6 5.5 0.06 Cascade combiner Expected model change Uncertainty sampling 0.64 10 0.74 3 6.5 0.05 Alternating combiner Expected model change Uncertainty sampling 0.69 2 0.72 12 7.0 0.05 Alternating combiner Query by committee Uncertainty sampling 0.64 11 0.72 8 9.5 0.05 Alternating combiner Expected model change Isolation forest ODAL 0.67 4 0.71 16 10.0 0.06 Weighted ranking combiner Query by committee Uncertainty sampling 0.65 9 0.72 11 10.0 0.05 Weighted ranking combiner Expected model change Isolation forest ODAL 0.64 13 0.72 10 11.5 0.04 Cascade combiner Uncertainty sampling Query by committee 0.62 20 0.72 7 13.5 0.06 Weighted ranking combiner Isolation forest ODAL Uncertainty sampling 0.64 14 0.71 14 14.0 0.05 Alternating combiner Expected model change Query by committee 0.65 8 0.70 20 14.0 0.05 Weighted ranking combiner Expected model change Query by committee 0.64 12 0.70 18 15.0 0.06 Cascade combiner Isolation forest ODAL Uncertainty sampling 0.63 16 0.71 15 15.5 0.05 Cascade combiner Uncertainty sampling Expected model change 0.61 22 0.72 9 15.5 0.05 Cascade combiner Expected model change Isolation forest ODAL 0.63 15 0.71 17 16.0 0.05 Cascade combiner Isolation forest ODAL Expected model change 0.60 24 0.71 13 18.5 0.05 Cascade combiner Isolation forest ODAL Query by committee 0.63 17 0.64 24 20.5 0.06 Weighted ranking combiner Isolation forest ODAL Query by committee 0.62 19 0.64 23 21.0 0.06 Alternating combiner Isolation forest ODAL Query by committee 0.62 18 0.63 25 21.5 0.07 Cascade combiner Query by committee Isolation forest ODAL 0.62 21 0.65 22 21.5 0.07 Cascade combiner Expected model change Query by committee 0.59 25 0.70 19 22.0 0.06 Cascade combiner Query by committee Expected model change 0.60 23 0.69 21 22.0 0.06 Table 10: Results comparison of policy combinations. 5.6. Large scale experiments 52 •Warm-up policies: – Isolation forest ODAL, is used both as the warm-up learner, but also as a single policy (after the random initialization). – Expected Model Change, Query By Committee and Uncertainty Sampling (w/ uncertainty defined as 0.5 threshold). To validate our three-phase framework we include two-phase experiments with these policies for comparison. •Hot policies: – Uncertainty Sampling with three different uncertainty definitions proposed in 3.6.4, respectively: the models scores closest to 0.5 ; the models scores closest to the fraud percentile and the epistemic uncertainty. –Query By Committee and Expected Model Change. •Switching criterion: – The switch between the cold and warm-up policy happens immediately after the first iteration (once the labeled pool is not empty). – Between the warm-up and the hot policy the switch happens when the first fraud is found (except for Query By Committee that requires at least two fraudulent instances). •The initial unlabeled pool size corresponds to one day of data. • Stopping criteria — the AL process stops running when seven days have passed (one day to train the preprocessing pipeline plus six additional days as, explained in section 3.5.1). The exception is the case of Bank 2, which has an extremely low number of positive instances, leading us to extend this period to 13 days. • Labelers — we use one simulated labeler that reviews 1000 transactions per day, which makes the final labeled pools always have 6000 transactions, except for Bank 2, which finishes with 13000 transactions. We can observe how the sequence of policies is specified in the results tables by inspecting one of the first ones (to be discussed in further detail). For example, in table 11 the first three columns display, respectively for every row, the cold, warm-up and hot phases/policies. When the keyword None is used, it means that the corresponding phase is absent, and the sequence of policies is for a one-phase or two-phase experiment. As stated in section 4.1, we will use five time-folds for each of the six different groups of experiments (for the various business cases explained in section 3.1): 1. Bank 1w/ card-not-present filter — the same use case presented in previous experiments but removing transactions that involve a payment with a physically present card. This filter is a common one in fraud detection, since fraud is often less probable when a card is present (we provide this new sub-case for comparison). 2. Bank 2— another bank, however with a much smaller fraud rate. This dataset does not have information on whether the transactions are with card present or not. 3. Merchant — with this dataset we have access to the types of labels, which allows us to also perform experiments where only real analysts’ feedback labels are used, as explained in section 4.1.1. 5.6. Large scale experiments 53 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 AVG rank all AVG VAR allCold policy Warm-up policy Hot policy Norm. Area Rank Norm. Area Rank Norm. Area Rank Norm. Area Rank Norm. Area Rank Random Isolation forest ODAL Uncertainty sampling (epistemic) 0.73 1 0.74 1 0.93 1 0.68 2 0.69 3 1.6 0.16 Random Isolation forest ODAL Uncertainty sampling (0.5) 0.73 2 0.74 2 0.93 2 0.69 1 0.69 4 2.2 0.16 Random None Uncertainty sampling (0.5) 0.72 3 0.73 4 0.91 5 0.67 3 0.68 5 4.0 0.18 Random Isolation forest ODAL None 0.54 5 0.67 5 0.90 7 0.56 6 0.71 2 5.0 0.13 Random Isolation forest ODAL Uncertainty sampling (fraud percentile) 0.55 4 0.74 3 0.91 3 0.55 7 0.52 10 5.4 0.26 Random Isolation forest ODAL Query by committee 0.53 7 0.59 6 0.81 9 0.57 5 0.61 6 6.6 0.21 Isolation forest outlier detection None None 0.36 11 0.53 11 0.90 6 0.52 8 0.71 1 7.4 0.19 Random None Query by committee 0.54 6 0.58 8 0.79 11 0.58 4 0.56 8 7.4 0.26 Random Isolation forest ODAL Expected model change 0.42 9 0.55 9 0.91 4 0.44 11 0.56 7 8.0 0.15 Random None None 0.48 8 0.58 7 0.81 10 0.50 9 0.52 11 9.0 0.26 Random None Expected model change 0.41 10 0.54 10 0.90 8 0.45 10 0.55 9 9.4 0.19 Table 11: Results’ comparison of the experiments with Bank 1CNP. 4.Merchant acquirer — this covers the last business case. The results tables to be discussed in these experiments (tables 11,12,14,13,15 and 16) follow a structure similar to the ones presented in previous sections: each row corresponds to a different candidate policy and the groups of columns with numerical results are for different time folds. For each dataset we present at least two example learning curve bands, (figures 17,18,20,23,25 and 26) which are the visualizations explained in section 4.4.1. The vertical axis is the target metric (e.g., recall at a target false-positive rate) and the horizontal axis is the size of the labeled pool. The plots are the intervals between percentiles of the target metric scores, so we can see the worst, median and best cases and the percentiles in-between. The horizontal red dashed line is the value of the full train data baseline, which is used to normalize the areas presented in the tables. In these analyzes, for each dataset, we present examples of the learning curves of the random policy and the best ranked policy (for that dataset) on the first time fold. In some cases, we present other learning curves to highlight other specific cases. 5.6.1Bank 1with card-not-present filter All the experiments presented in the previous chapters were performed on a dataset from the banking use case. Frequently, in this business case, the goal is to classify transactions where the physical card was not present, such as online payments. This usually results in a dataset with a larger fraud rate (section 4.1). Therefore, to cover this sub-use case, we apply a CNP filter. In table 11 we present the results of the experiments for this use case, from which we can conclude the following: 5.6. Large scale experiments 54 0 1000 2000 3000 4000 5000 6000 Labeled pool size 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Target metric Percentile 50 Percentile 0 - 100 Percentile 16 - 83 Percentile 33 - 66 Full train data (a) Random policy 0 1000 2000 3000 4000 5000 6000 Labeled pool size 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Target metric Percentile 50 Percentile 0 - 100 Percentile 16 - 83 Percentile 33 - 66 Full train data (b) Three-phase epistemic uncertainty sampling. Figure 17: Performance in the first fold of Bank 1with the random policy and the best ranked policy. • the various uncertainty sampling implementations and the standalone isolation forest ODAL, similar to the results presented in earlier sections for this dataset without CNP filter, are the best performing methods. •every hot policy in a 3-phase setup outperforms its two-phase counterpart. • the learning curves for this dataset have a relatively low variance, in particular the best performing policies (see also figure 17). • epistemic uncertainty sampling outperforms all other methods in most folds, achieving relatively low values of variance, (though the differences are very small, and likely not significant, compared to other uncertainty measures). In figure 17 we can see two example learning curve plots for these experiments: random policy (left panel) and epistemic uncertainty sampling (right panel), for the first fold. We can observe that the random policy has much more variance up to the 3500 instances. Furthermore, it plateaus at a considerably smaller central value close to 0.2 . On the other hand, the best ranked policy has a significantly smaller variance and it reaches the full train data baseline performance near the end of the run (with a central value close to 0.3). As expected the results do not differ much from the Bank 1dataset without CNP filter used in earlier sections. Since the results are consistent across the five time folds, the study for this dataset suggests that the usage of the three-phase epistemic uncertainty sampling policy is an appropriate choice for banking datasets with fraud rates similar to this one. 5.6.2Bank 2 The dataset for Bank 2represents a great challenge even when training a ML model in an environment without AL, due to its extremely low fraud rate of 0.03% . Because of this very low rate, instead of running experiments only for seven days we extend it to two weeks. Hence, the maximum labeled 5.6. Large scale experiments 55 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 AVG rank all AVG VAR allCold policy Warm-up policy Hot policy Norm. Area Rank Norm. Area Rank Norm. Area Rank Norm. Area Rank Norm. Area Rank Random Isolation forest ODAL Uncertainty sampling (epistemic) 0.16 5 0.54 1 0.55 3 0.33 3 0.55 2 2.8 0.47 Random Isolation forest ODAL Uncertainty sampling (0.5) 0.16 4 0.53 2 0.50 4 0.35 1 0.43 7 3.6 0.45 Random Isolation forest ODAL Uncertainty sampling (fraud percentile) 0.17 2 0.52 3 0.49 5 0.33 4 0.44 5 3.8 0.47 Random None Uncertainty sampling (0.5) 0.15 6 0.35 8 0.48 6 0.33 2 0.59 1 4.6 0.54 Random Isolation forest ODAL Expected model change 0.11 9 0.48 4 1.05 1 0.18 9 0.44 6 5.8 0.31 Random None None 0.20 1 0.34 9 0.26 9 0.27 6 0.39 8 6.6 0.47 Random None Expected model change 0.14 7 0.31 11 0.59 2 0.12 10 0.54 3 6.6 0.57 Random Isolation forest ODAL None 0.17 3 0.42 7 0.25 10 0.21 7 0.31 10 7.4 0.23 Random Isolation forest ODAL Query by committee 0.12 8 0.42 6 0.34 7 0.21 8 0.32 9 7.6 0.32 Random None Query by committee 0.11 11 0.31 10 0.33 8 0.31 5 0.52 4 7.6 0.46 Isolation forest outlier detection None None 0.11 10 0.45 5 0.22 11 0.10 11 0.15 11 9.6 0.16 Table 12: Results of the experiments with Bank 2. pool size is then 13000, corresponding to a period of 1day to collect data for pre-processing and 13 days for labeling. The results in table 12 lead us to the following observations: • Most AL experiments with this dataset have very noisy learning curves when compared to Bank 1, since the values of the average variance are much larger. • The normalized areas under the median learning curve, especially in folds 1and 4, are very low, indicating that these AL experiments are far under-performing the full train data baseline. • Exceptionally for fold 1, the random policy is the best performer because its high variance gives it a chance to reach larger values (while other policies struggle to learn any useful pattern to select queries). This behaviour is clear in figure 18. Hence, even though in this fold it often performs better, it is very unreliable. In contrast, for this fold, epistemic uncertainty is always in a plateau between 0 and 0.1. • In fold 3, the three-phase Expected Model Change obtains a much higher normalized area when compared to the other policies. We can see the comparison of the learning curves of this policy with the overall best ranked policy (epistemic uncertainty sampling) in figure 19. We can observe that, the epistemic uncertainty sampling has a regular growth throughout the process and reaches at least the full train data baseline for a large fraction of the learning curves. On the other hand, the policy with Expected Model Change (on the right) tends to jump to a value above the full train data baseline when the labeled pool has a size around 3000 transactions. •the best ranked policy is, once again, epistemic uncertainty sampling. Regarding the full train data baseline, it is substantially larger than most AL results (see, e.g., figure 18 where its value is around 0.43 ) – with the exception of the examples presented in figure 19 5.6. Large scale experiments 56 0 2000 4000 6000 8000 10000 12000 Labeled pool size 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Target metric Percentile 50 Percentile 0 - 100 Percentile 16 - 83 Percentile 33 - 66 Full train data (a) Random policy 0 2000 4000 6000 8000 10000 12000 Labeled pool size 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Target metric Percentile 50 Percentile 0 - 100 Percentile 16 - 83 Percentile 33 - 66 Full train data (b) Three-phase epistemic uncertainty sampling. Figure 18: Performance in the first fold of Bank 2with the random policy and the best average rank policy. 0 2000 4000 6000 8000 10000 12000 Labeled pool size 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Target metric Percentile 50 Percentile 0 - 100 Percentile 16 - 83 Percentile 33 - 66 Full train data (a) Three-phase epistemic uncertainty sampling. 0 2000 4000 6000 8000 10000 12000 Labeled pool size 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Target metric Percentile 50 Percentile 0 - 100 Percentile 16 - 83 Percentile 33 - 66 Full train data (b) Three-phase Expected Model Change. Figure 19: Performance in the third fold of Bank 2with the best overall ranked policy (left figure) and an odd result (right figure). 5.6. Large scale experiments 57 and table 12, where we observe some higher performance values near the end of the run. This suggests that we may be able to obtain a higher performance, i.e., more comparable to Bank 1, by increasing the daily review budget to collect more labels. Overall, this means that with such a conservative analyst budget (corresponding to a single analyst reviewing 1000 transactions per day) AL is not always able to deal with such an extreme class imbalance. By increasing the review budget we may be able to decrease the noise in the learning curves and obtain better results. 5.6.3Merchant As explained in section 4.1.1, the Merchant dataset has the advantage of containing details on how each label was obtained. We exploit this in order to perform different sets of experiments with datasets containing labels only obtained through: •Analysts’ feedback — transactions reviewed by analysts. • Confident analysts’ feedback — transactions that analysts have reviewed and stated they were very confident of the assigned label. • Chargebacks — transactions that a client has complained about. Due to its nature, this is the type of label that is less likely to contain mistakes. Analysts’ feedback only This scenario, allows us to only use transactions with labels assigned by an analyst to simulate a realistic case of an analyst who can make some mistakes. In order to produce an evaluation that is as robust as possible, we also use chargebacks in the test set, since they are the most robust labels and may have a label that is opposite to what an analyst would have decided in the AL process. In table 13 we present the results for this scenario, where the following can be observed: • In the first fold, the normalized areas do not differ much between different policies, even when using the random policy, which indicates that, for this exceptional case, the policy choice is not very important. This is especially clear when observing the learning curves of the random policy and the overall best ranked policy (in this dataset) in figure 20. Both are very similar and have very low variance, achieving a very stable plateau at the full train data baseline. We speculate that this could indicate that the fraud patterns in this train fold are sufficiently varied that we are collecting a representative sample by just randomly sampling, whereas for other folds there could be a dominant pattern and then AL helps finding the complementary ones faster. Another possibility is that the test set contains only transactions that are very easy to label. • There are normalized areas much above 1 (e.g., most policies in fold 4), hence, the AL runs are outperforming the full train data baseline. In figure 21 we present two learning curves on the 4 th time fold, the random policy and the one with the largest normalized area (epistemic uncertainty sampling). The random policy grows with a regular slope throughout the process. On the other hand the very high performance epistemic uncertainty sampling curve (right 5.6. Large scale experiments 58 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 AVG rank all AVG VAR allCold policy Warm-up policy Hot policy Norm. Area Rank Norm. Area Rank Norm. Area Rank Norm. Area Rank Norm. Area Rank Random Isolation forest ODAL Uncertainty sampling (0.5) 0.93 2 1.02 2 0.79 2 2.24 3 1.36 3 2.4 0.35 Random None Uncertainty sampling (0.5) 0.93 3 1.02 1 0.79 3 2.29 2 1.35 4 2.6 0.36 Random Isolation forest ODAL Uncertainty sampling (epistemic) 0.90 5 0.88 4 0.79 1 2.33 1 1.11 5 3.2 0.29 Random None Expected model change 0.89 8 0.89 3 0.74 9 1.94 6 1.54 1 5.4 0.39 Random Isolation forest ODAL Expected model change 0.89 9 0.86 5 0.74 8 1.96 5 1.52 2 5.8 0.39 Random Isolation forest ODAL None 0.91 4 0.60 7 0.75 7 1.52 7 0.66 7 6.4 0.30 Isolation forest outlier detection None None 0.90 6 0.72 6 0.69 10 1.97 4 0.78 6 6.4 0.31 Random None None 0.90 7 0.45 8 0.77 6 0.87 8 0.62 8 7.4 0.39 Random None Query by committee 0.88 10 0.10 10 0.78 4 0.30 9 0.38 9 8.4 0.16 Random Isolation forest ODAL Query by committee 0.88 11 0.11 9 0.78 5 0.30 10 0.37 10 9.0 0.17 Random Isolation forest ODAL Uncertainty sampling (fraud percentile) 0.96 1 0.09 11 0.24 11 0.25 11 0.37 11 9.0 0.23 Table 13: Results of the experiments with the Merchant using only analysts’ feedback. panel), plateau, in most cases, at around 2000 transactions with a target metric central value close to 3times the full train data baseline value. • In all folds, AL is either providing a clear advantage over random sampling (folds 2,4and 5), or it is at least as good. • Uncertainty sampling with the uncertainty target as the threshold 0.5is the overall best ranked policy. We conjecture two possible reasons to blame for the very low performance of the full train data baseline in some folds: • Since we are using the same model as for AL, it might not be appropriate for training with so much more data and therefore ends up under-fitting it. • The baseline model is trained with data extending over a wider period (i.e., the full train data baseline has access to two weeks of data, whereas AL only uses a sample from the first week), so its performance could be harmed by the existence of noise, on the second week, that is not present in the AL labeled pools. The (likely) best solution would be to perform hyper-parameter tuning and maybe even model selection to the process of computing the full train data baseline. Done properly, this would ensure the best ML model configuration for (the given data) is always being used and would result in a full train data baseline that actually represents a full train data baseline for the given train/test split. 5.6. Large scale experiments 59 0 1000 2000 3000 4000 5000 6000 Labeled pool size 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Target metric Percentile 50 Percentile 0 - 100 Percentile 16 - 83 Percentile 33 - 66 Full train data (a) Random policy 0 1000 2000 3000 4000 5000 6000 Labeled pool size 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Target metric Percentile 50 Percentile 0 - 100 Percentile 16 - 83 Percentile 33 - 66 Full train data (b) Three-phase uncertainty sampling (using 0.5 threshold). Figure 20: Performance in the first fold of the Merchant with analyst feedback only using the random policy and the best ranked policy. 0 1000 2000 3000 4000 5000 6000 Labeled pool size 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Target metric Percentile 50 Percentile 0 - 100 Percentile 16 - 83 Percentile 33 - 66 Full train data (a) Random policy. 0 1000 2000 3000 4000 5000 6000 Labeled pool size 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Target metric Percentile 50 Percentile 0 - 100 Percentile 16 - 83 Percentile 33 - 66 Full train data (b) Three-phase epistemic uncertainty sampling. Figure 21: Performance in the 4 th fold of the merchant dataset with analysts’ feedback only, with the random policy (left figure) and an odd result (right figure), due to having a very high normalized area (2.23). 5.6. Large scale experiments 60 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 AVG rank all AVG VAR allCold policy Warm-up policy Hot policy Norm. Area Rank Norm. Area Rank Norm. Area Rank Norm. Area Rank Norm. Area Rank Random Isolation forest ODAL Uncertainty sampling (epistemic) 0.94 2 0.68 3 0.79 1 2.02 1 0.68 5 2.4 0.26 Random None Uncertainty sampling (0.5) 0.93 3 0.81 2 0.78 2 1.96 3 0.85 3 2.6 0.31 Random Isolation forest ODAL Uncertainty sampling (0.5) 0.93 4 0.82 1 0.78 3 1.96 2 0.84 4 2.8 0.31 Random None Expected model change 0.91 8 0.64 5 0.74 8 1.47 4 1.16 1 5.2 0.35 Random Isolation forest ODAL Expected model change 0.91 7 0.66 4 0.74 9 1.41 5 1.13 2 5.4 0.34 Random Isolation forest ODAL None 0.92 5 0.32 7 0.75 7 1.07 7 0.39 7 6.6 0.26 Random None None 0.89 9 0.29 8 0.78 4 0.47 8 0.42 6 7.0 0.33 Isolation forest outlier detection None None 0.92 6 0.40 6 0.71 10 1.27 6 0.38 8 7.2 0.25 Random Isolation forest ODAL Uncertainty sampling (fraud percentile) 0.96 1 0.07 9 0.44 11 0.18 11 0.27 11 8.6 0.21 Random None Query by committee 0.87 10 0.07 10 0.77 6 0.23 10 0.27 9 9.0 0.14 Random Isolation forest ODAL Query by committee 0.87 11 0.07 11 0.77 5 0.24 9 0.27 10 9.2 0.13 Table 14: Results of the experiments with the Merchant using only confident analysts’ feedback. Confident analysts’ feedback only In this experiment we only provide labels that have been reviewed by analysts and were assigned with a high level of confidence. This allows us to understand if using only labels that should have lower levels of noise is beneficial in AL. The results of this experiment are presented in table 14, from which we conclude the following: • Just like when using only analysts’ feedback, the overall results are good. Fold 4also has extreme values. The main differences are that the normalized areas are lower in fold 2and 5and that, here, the best ranked policy is epistemic uncertainty sampling instead of the 0.5 threshold uncertainty method. Note, however, that if we look at the numbers, the ranking of these two policies would be swapped if we used an average of the KPI instead of an average ranking. This is because the absolute differences in the KPI values are, in many cases, small, so the differences in the ranks do not have significance. • The best ranked policy is the epistemic uncertainty sampling, but note that overall, the differences in average ranks of the top three policies are not very significant. In order to understand the differences in the normalized areas, in figure 22 we can see a comparison between the two datasets with the same policy in fold two. We can observe that there are two main reasons why the normalized areas are inferior when using only confident labels: • the median target metric score at plateau, when using only confident feedback (approximately 0.55), is lower than when using all feedback (approximately 0.65) • the full train data baseline score is larger, which makes the normalization term larger, hence lowering the KPI. 5.6. Large scale experiments 67 Bank 1 Bank 2 Merchant w/ analysts feedback Merchant w/ confident analysts feedback Merchant w/ chargebacks Merchant acquirer AVG Cold policy Warm-up policy Hot policy Random Isolation forest ODAL Uncertainty sampling (epistemic) 1 1 3 1 1 4 1.8 Random Isolation forest ODAL Uncertainty sampling (0.5) 2 2 1 3 2 1 1.8 Random None Uncertainty sampling (0.5) 3 4 2 2 3 2 2.7 Random Isolation forest ODAL Expected model change 9 5 5 5 5 3 5.3 Random None Expected model change 11 6 4 4 7 4 6.0 Random Isolation forest ODAL None 4 8 6 6 9 7 6.7 Random Isolation forest ODAL Uncertainty sampling (fraud percentile) 5 3 10 9 7 9 7.2 Random None None 10 6 8 7 9 8 8.0 Isolation forest outlier detection None None 7 11 6 8 11 6 8.2 Random Isolation forest ODAL Query by committee 6 9 10 11 4 10 8.3 Random None Query by committee 7 9 9 10 6 11 8.7 Table 17: Every policy used in the large-scale experiments and their ranks in the datasets. The policies are sorted by average rank. 6 CONCLUSIONS In this dissertation, we studied active learning (AL) methods to build a ML model for fraud detection. We explored a realistic setup of a new system where there is no historical data and the system needs to be deployed to collect data and start building a useful ML model. We designed a versatile system architecture (Section 3.4) that allows us to deploy AL in a live production system. In designing this solution we took into account: •how to perform pre-processing to prepare the data for the AL framework. • how to manage the data, which is incoming from a live data stream to be stored in an unlabeled data pool and is then gradually sampled to create a labeled data pool. • that it iteratively selects batches of data instances to be labeled when configured with a given set of AL policies and switching criteria between policies. • it contains a component that, given a team of labelers (i.e., the analysts/oracles) and a distribution schedule, queries the analysts for the chosen transactions’ labels. • the possibility to train ML models at any AL iteration in any way, not only for deployment but also to perform evaluations of the current system performance. Then, we extended this architecture to be able to conduct simulation experiments of a production system using historical datasets (Section 3.5). This includes: • the implementation of a data stream to process transactions in a time ordered fashion and to insert transactions in the labeled pool according to the time that analysts take to review the batches; • the development of a simulated labeler that uses the real labels of the transactions (since we used a historical dataset) and spends a given time labeling a transaction; • the development of an evaluation component that can train ML models on the labeled pool and make predictions on a test set throughout the AL run to measure its performance. When evaluating our experiments (Chapter 4), for each policy we performed experiments on several time folds. A time fold is defined as a period of 4weeks with the first two weeks available for AL and the other two used as a test set. In practice we only used the first week for AL except for one dataset with an extremely large class imbalance (for which we used the two weeks). We introduced a visualization technique to be able to analyze the behavior of the learning curves of the AL runs, as well as Key Performance Indicators (KPIs) based on these learning curves. This allows us to quantify the performance of an AL configuration and compare several policies or time folds. 68 69 The in-depth literature review on the current state-of-the-art in the AL field (Chapter 2), led us to a set of policies to experiment with, as well as to our custom made methods. We implemented (Section 3.6): •Expected model change. • Uncertainty sampling, which queries instances that obtain scores closer to 0.5 . We also implemented our own two variants of this method: the fraud percentile uncertainty sampling and epistemic uncertainty sampling. • Query by committee with our own modification on how the model disagreement measure is computed. • A density-weighted method that turned out to be too computationally heavy for fraud detection due to the large dataset sizes (therefore discarded). •Our novel Outlier Discriminative Active Learning (ODAL) method. We launched a set of preliminary experiments (Section 5.1), with only two time folds of a dataset, that led us to the conclusion that uncertainty sampling and isolation forest ODAL were performing better. We also performed a set of experiments to validate the data pre-processing method used in the preliminary experiment. This assists in the decision on which method should be used in the final large scale experiments for policy comparison (Section 5.2). The main candidates were: • Feedzai’s AutoML tool to generate many features based on the user profiles and then apply either: i) feature reduction with Principal Components Analysis (PCA) or ii) pairwise correlation selection. • Feedzai’s AutoML tool to generate less features already using expert knowledge to manually narrow down the number of profiles. From the results of the experiments we concluded that generating many features with the AutoML tool and then reducing them with PCA was the best alternative. We also performed a similar set of experiments to validate ML model options and decide which one we should be using further on (Section 5.3). From several ML models, each with varying sets of hyper-parameters, we concluded that: • For a given fixed model type, varying the hyper-parameters in a way that increases regularization increases performance, as expected given the small sample sizes produced by AL. •The random forest models turned out to be the most effective candidates. In light of these preliminary experiments, we then moved on to observing that one of the main challenges of applying AL in fraud detection is the extreme class imbalance of some datasets. Given that most AL policies require the labeled pool to contain labels of all classes, this sometimes results in AL runs that only switch to this hot policy in very late stages of the process. The standard approach in the AL literature is to produce an initial random sample until the hot policy can be used, which often results in a later switching. We introduced a solution for this problem (Section 3.7): a three-phase framework where there is a cold, a warm-up and a hot policy. In all later experiments (Section 5.4 70 and 5.6) we concluded that, for any hot policy, its performance in the three-phase framework (with ODAL as warm-up policy) is always better than a two-phase framework. Another hypothesis we considered was that the combination of two AL policies could perform better than each separate policy. In order to understand the potential of combining policies, we developed policy divergence diagnostics that can assess how different two AL policies are (Section 3.8.1): cluster frequency divergence and model disagreement divergence. We performed experiments using these diagnostic metrics (Section 5.5.1) and concluded that, as expected, between pair of policies that are intrinsically similar (e.g., elliptic envelope ODAL and isolation forest ODAL) they signal very low levels of divergence whereas for some of the other pairs higher values were found. This is a good indication that the diagnostics metrics work as they should. However, while the cluster frequency divergence indicated that some policies could definitely benefit from combination, the model disagreement divergence results were not so clear about such benefit. This hints that, from a performance point of view, the benefits of the combination would be smaller. In order to combine two policies, we then introduced three methods (Section 3.8.2): the weighted ranking combiner, the alternating combiner and the cascade combiner; In the experiments performed with these combiners (Section 5.5) we concluded that none of the combined policies outperformed the best performing single policy (uncertainty sampling). Thus, we concluded that it was not worth including policy combination in the final large scale experiments. Finally, armed with a good understanding of which AL configurations should be further investigated, we presented the final large scale experiments on various datasets. For these, we introduced four datasets (Section 4.1): Bank 1with a card-not-present (CNP) filter, Bank 2with a very small fraud rate, a Merchant that processes payments from its consumers and a Merchant acquirer that processes payments for several merchants from their consumers. The Merchant dataset contained further information on how each of the labels was obtained. This allowed us to generate three variations of this dataset that used only: •all analysts’ feedback (allowing us to simulate the effect of the analysts’ mistakes). • confident analysts’ feedback (to allow us to understand if only using labels marked as confident could reduce the negative impact introduced by label noise) •chargeback labels (from direct client complaints, which should have low label noise). With everything set, we launched the final large-scale experiments and analyzed their results (Section 5.6). The conclusions were as follows: • AL appears to be an appropriate solution for datasets similar to Bank 1. It always reaches the full train data baseline performance and the levels of variance are considerably lower than the random policy. This means that, in this domain, the empirical results indicate that a good model can be obtained with a small budget of queries. • With datasets that have very low fraud rates, like bank 2, AL does not bring very stable results with the budget size we used. • The Merchant, for which we generated three different types of datasets for different label scenarios, led us to understand that using only chargebacks (the scenario that should have much less noise) does not provide stable results. On the other hand using analysts’ feedback 71 we do obtain good results, with minor differences between using only confident feedback or all feedback. • The Merchant acquirer dataset, just like Bank 2, had several poor results that still need to be understood to find further improvements, before using AL in this business case. • In a meta-analysis presented in table 17, to combine the policy rankings from different datasets, we concluded that the best overall ranked set of policies was our proposed three-phase setup with isolation forest ODAL as a warm-up policy and uncertainty sampling (either epistemic or with threshold at 0.5) as hot policy. The conclusions presented in this study open up several further questions that are left for future work. Namely: • There were a few experiments/folds for Bank 2, for the Merchant with chargebacks only and for the Merchant acquirer with anomalous learning curves or optimistic baselines with lower performance. Trying to understand the reasons behind such issues and finding ways to overcome them would be an important next step. • In particular for Bank 2, one of the reasons for those issues could be simply related to the extremely low fraud rate (which results in a very large variance). If that is the case, experiments with a higher budget of analysts should be performed to determine if better results can be obtained that are more similar to Bank 1. • For the Merchant dataset with chargebacks only, contrarily to the datasets using analysts’ feedback, the test set only had chargebacks. This difference may have harmed the results, because there is less fraud labels in chargebacks. Redoing the evaluations with the same labelling setup on the test set would make the three evaluations more directly comparable. • In all the experiments, we trained models using the exact same model and hyper-parameters when the labeled pool has 50 or 6000 instances and we also do not adapt them according to the datasets. Performing hyper-parameters tuning in AL and when computing the full train data baselines could have an impact on our evaluation. This could also be helpful to address the issues mentioned above. • One increasing concern in the fraud detection field is money laundering. This is a domain for which it is harder to build a ML model since, unlike with other types of fraud, there are no client complaints. This results in a label scarcity problem. As already demonstrated in Lorenz et al. (2020), where the tools developed in our work were used, this is a field where AL seems to be the most promising strategy. Therefore, further research on the usage of AL for anti-money laundering could bring interesting results. • As typically done in fraud detection, transactions for which there is no feedback (i.e., there were no complaints but no analyst have reviewed them either) are presumed to be legitimate. This could be a source of label noise and using AL may help find those instances and bring value by reducing label noise. • Our experiments resorted to a simulated analyst. Performing experiments with actual human analysts is an important next step since it might either reveal limitations or suggest adjustments, which may be necessary to apply to our proposed AL methods to make them successful. BIBLIOGRAPHY Jo ˜ ao Tiago Ascens ˜ ao, Pedro Bizarro, Ricardo Barata, Miguel Leite, Ricardo Pacheco, and Marco O.P. Sampaio. Active learning for online training in imbalanced data streams under cold start. Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng, and J ¨ org Sander. Lof: Identifying densitybased local outliers. SIGMOD Rec.,29(2):93–104, May 2000. ISSN 0163-5808. doi: 10.1145/335191. 335388. URL https://doi.org/10.1145/335191.335388. Fabrizio Carcillo, Yann-A ¨ el Le Borgne, Olivier Caelen, and Gianluca Bontempi. Streaming active learning strategies for real-life credit card fraud detection: assessment and visualization. International Journal of Data Science and Analytics,5(4):285–300,2018. Weijie Fu, Meng Wang, Shijie Hao, and Xindong Wu. Scalable active learning by approximated error reduction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’18, pages 1396–1405, New York, NY, USA, 2018. ACM. ISBN 978-1-45035552-0. doi: 10.1145/3219819.3219954. URL http://doi.acm.org/10.1145/3219819.3219954. Jo ˜ ao Gama, Indr ˙ e ˇ Zliobait ˙ e, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey on concept drift adaptation. ACM computing surveys (CSUR),46(4):44,2014. Daniel Gissin and Shai Shalev-Shwartz. Discriminative active learning. CoRR, abs/1907.06347,2019. URL http://arxiv.org/abs/1907.06347. S. Huang, R. Jin, and Z. Zhou. Active learning by querying informative and representative examples. IEEE Transactions on Pattern Analysis and Machine Intelligence,36(10):1936–1949, Oct 2014. ISSN 1939-3539. doi: 10.1109/TPAMI.2014.2307881. Yufeng Kou, Chang-Tien Lu, Sirirat Sirwongwattana, and Yo-Ping Huang. Survey of fraud detection techniques. In IEEE International Conference on Networking, Sensing and Control, 2004, volume 2, pages 749–754. IEEE, 2004. Kenneth Lang and Eric Baum. Query learning can work poorly when a human oracle is used, 1992. David D. Lewis and Jason Catlett. Heterogeneous uncertainty sampling for supervised learning. In William W. Cohen and Haym Hirsh, editors, Machine Learning Proceedings 1994, pages 148 – 156. Morgan Kaufmann, San Francisco (CA), 1994. ISBN 978-1-55860-335-6. doi: https://doi. org/10.1016/B978-1-55860-335-6.50026-X. URL http://www.sciencedirect.com/science/article/pii/ B978155860335650026X. X. Li and Y. Guo. Adaptive active learning for image classification. In 2013 IEEE Conference on Computer Vision and Pattern Recognition, pages 859–866, June 2013. doi: 10.1109/CVPR.2013.116. Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory,37(1):145–151,1991. 72 bibliography 73 Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In Proceedings of the 2008 Eighth IEEE International Conference on Data Mining, ICDM ’08, page 413–422, USA, 2008. IEEE Computer Society. ISBN 9780769535029. doi: 10.1109/ICDM.2008.17. URL https://doi.org/10.1109/ICDM.2008.17. Joana Lorenz, Maria In ˆ es Silva, David Apar ´ ıcio, Jo ˜ ao Tiago Ascens ˜ ao, and Pedro Bizarro. Machine learning methods to detect money laundering in the bitcoin blockchain in the presence of label scarcity. arXiv preprint arXiv:2005.14635,2020. Paulo C ´ esar Gon c¸ alves Marques, Miguel Ramos de Ara ´ ujo, Bruno Casal Lara ˜ na, Nuno Miguel Louren c¸ o Diegues, Pedro Cardoso Lessa e Silva, and Pedro Gustavo Santos Rodrigues Bizarro. Semantic-aware feature engineering, March 19 2020. US Patent App. 16/567,761. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research,12: 2825–2830,2011. F ´ abio Pinto, Marco OP Sampaio, and Pedro Bizarro. Automatic model monitoring for data streams. arXiv preprint arXiv:1908.04240,2019. Kedar Potdar, Taher S Pardawala, and Chinmay D Pai. A comparative study of categorical variable encoding techniques for neural network classifiers. International journal of computer applications,175 (4):7–9,2017. Peter J. Rousseeuw and Katrien Van Driessen. A fast algorithm for the minimum covariance determinant estimator. Technometrics,41(3):212–223,1999. doi: 10.1080/00401706.1999.10485670. URL https://www.tandfonline.com/doi/abs/10.1080/00401706.1999.10485670. Nicholas Roy and Andrew McCallum. Toward optimal active learning through sampling estimation of error reduction. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, pages 441–448, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1-55860-778-1. URL http://dl.acm.org/citation.cfm?id=645530.655646. Bernhard Sch ¨ olkopf, Robert Williamson, Alex Smola, John Shawe-Taylor, and John Platt. Support vector method for novelty detection. In Proceedings of the 12th International Conference on Neural Information Processing Systems, NIPS’99, page 582–588, Cambridge, MA, USA, 1999. MIT Press. Burr Settles. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin–Madison, 2009. H. S. Seung, M. Opper, and H. Sompolinsky. Query by committee. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory, COLT ’92, pages 287–294, New York, NY, USA, 1992. ACM. ISBN 0-89791-497-X. doi: 10.1145/130385.130417. URL http://doi.acm.org/10.1145/130385. 130417. Mohammad Hossein Shaker and Eyke H ¨ ullermeier. Aleatoric and epistemic uncertainty with random forests. In International Symposium on Intelligent Data Analysis, pages 444–456. Springer, 2020. bibliography 74 Asli Uyar, Ayse Bener, H Nadir Ciray, and Mustafa Bahceci. A frequency based encoding technique for transformation of categorical variables in mixed ivf dataset. In 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pages 6214–6217. IEEE, 2009. Andreas Vlachos. A stopping criterion for active learning. Computer Speech & Language,22(3):295–312, 2008. Min Wang, Fan Min, Zhi-Heng Zhang, and Yan-Xue Wu. Active learning through density clustering. Expert Systems with Applications,85:305 –317,2017. ISSN 0957-4174. doi: https://doi.org/10.1016/j. eswa.2017.05.046. URL http://www.sciencedirect.com/science/article/pii/S095741741730369X. Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems,2(1-3):37–52,1987. Rui Xu and Donald Wunsch. Survey of clustering algorithms. IEEE Transactions on neural networks,16 (3):645–678,2005. Yazhou Yang and Marco Loog. A benchmark and comparison of active learning for logistic regression, 2016. Lei Yu and Huan Liu. Feature selection for high-dimensional data: A fast correlation-based filter solution. In Proceedings of the 20th international conference on machine learning (ICML-03), pages 856–863,2003. Yifan Zhang, Peilin Zhao, Shuaicheng Niu, Qingyao Wu, Jiezhang Cao, Junzhou Huang, and Mingkui Tan. Online adaptive asymmetric active learning with limited budgets. IEEE Transactions on Knowledge and Data Engineering,2019. Jingbo Zhu, Huizhen Wang, Eduard Hovy, and Matthew Ma. Confidence-based stopping criteria for active learning for data annotation. ACM Transactions on Speech and Language Processing (TSLP),6 (3):3,2010a. Xingquan Zhu, Peng Zhang, Xiaodong Lin, and Yong Shi. Active learning from stream data using optimal weight classifier ensemble. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics),40(6):1607–1621,2010b. Indr ˙ e ˇ Zliobait ˙ e, Albert Bifet, Bernhard Pfahringer, and Geoff Holmes. Active learning with evolving streaming data. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 597–612. Springer, 2011.