A scalable and automated machine learning framework to support risk management

A Scalable and Automated Machine Learning Framework to Support Risk Management Lu´ıs Ferreira1,2[0000−0002−4790−5128], Andr´e Pilastri2[0000−0002−4380−3220], Carlos Martins3[0000−0002−0678−4868], Pedro Santos3[0000−0002−4269−5838], and Paulo Cortez2[0000−0002−7991−2090] 1EPMQ - IT Engineering Maturity and Quality Lab, CCG ZGDV Institute, Guimar˜aes, Portugal {luis.ferreira, andre.pilastri}@ccg.pt 2ALGORITMI Centre, Dep. Information Systems, University of Minho, Guimar˜aes, Portugal [email protected] 3WeDo Technologies, Braga, Portugal {pedro.santos, carlos.mmartins}@mobileum.com Abstract. Due to the growth of data and widespread usage of Machine Learning (ML) by non-experts, automation and scalability are becoming key issues for ML. This paper presents an automated and scalable framework for ML that requires minimum human input. We designed the framework for the domain of telecommunications risk management. This domain often requires non-ML-experts to continuously update supervised learning models that are trained on huge amounts of data. Thus, the framework uses Automated Machine Learning (AutoML), to select and tune the ML models, and distributed ML, to deal with Big Data. The modules included in the framework are task detection (to detect classification or regression), data preprocessing, feature selection, model training, and deployment. In this paper, we focus the experiments on the model training module. We first analyze the capabilities of eight AutoML tools: Auto-Gluon, Auto-Keras, Auto-Sklearn, Auto-Weka, H2O AutoML, Rminer, TPOT, and TransmogrifAI. Then, to select the tool for model training, we performed a benchmark with the only two tools that address a distributed ML (H2O AutoML and TransmogrifAI). The experiments used three real-world datasets from the telecommunications domain (churn, event forecasting, and fraud detection), as provided by an analytics company. The experiments allowed us to measure the computational effort and predictive capability of the AutoML tools. Both tools obtained high-quality results and did not present substantial predictive differences. Nevertheless, H2O AutoML was selected by the analytics company for the model training module, since it was considered a more mature technology that presented a more interesting set of features (e.g., integration with more platforms). After choosing H2O AutoML for the ML training, we selected the technologies for the remaining components of the architecture (e.g., data preprocessing and web interface). Keywords: Automated Machine Learning ·Distributed Machine Learning ·Supervised Learning ·Risk Management. 2 Lu´ıs Ferreira et al. 1 Introduction Nowadays, Machine Learning applications can make use of a great amount of data, complex algorithms, and machines with great processing power to produce effective predictions and forecasts [11]. Currently, two of the most important features of real-world ML applications are distributed learning and AutoML. Distributed learning is particularly useful for ML applications in the context of Big Data or when there are hardware constraints. Distributed learning consists of using multiple machines or processors to process parts of the ML algorithm or parts of the data. The fact that it is possible to add new processing units enables ML applications to surpass time and memory restrictions [29]. AutoML intends to allow people that are not experts in ML to efficiently choose and apply ML algorithms. AutoML is particularly relevant since there is a growing number of non-specialists working with ML [31]. It is also important for realworld applications that require constant updates to ML models. In this paper, we propose a technological architecture that addresses these two ML challenges. The architecture was adapted to the area of telecommunications risk management, which is a domain that mostly uses supervised learning algorithms (e.g., for churn prediction). Moreover, the ML models are constantly updated by people that are not experts in ML and may involve Big Data. Thus, the proposed architecture delineates a set of steps to automate the typical workflow of a ML application that uses supervised learning. The architecture includes modules for task detection, data preprocessing, feature selection, model training, and deployment. The focus of this work is the model training module of the architecture, which was designed to use a distributed AutoML tool. In order to select the ML tool for this module, we initially evaluated the characteristics of eight opensource AutoML tools (Auto-Gluon, Auto-Keras, Auto-Sklearn, Auto-Weka, H2O AutoML, Rminer, TPOT, and TransmogrifAI). We then performed a benchmark to compare the two tools that allowed a distributed execution (H2O AutoML and TransmogrifAI). The experiments used three real-world datasets from the domain of telecommunications. These datasets were related to churn (regression), event forecasting (time series), and fraud detection (binary classification). This paper consists of an extended version of our previous work [14]. The main novelty of this extended version is the technological architecture that is presented in Section 6. This section describes the particular technologies that were used to implement the components of the proposed AutoML distributed framework apart from model training. Also, this section describes the REST API that was developed to mediate the communication between the end-users and the proposed framework. The paper is organized as follows. Section 2 presents the related work. In Section 3, we detail the proposed ML architecture. Nest, Section 4 describes the analyzed AutoML technologies and the datasets used during the experimental tests. Then, Section 5 discusses the experimental results. Section 6 details the technological architecture. Finally, Section 7 presents the main conclusions and future work directions. A Scalable and Automated ML Framework 3 2 Related Work In a Big Data context, it is critical to create and use scalable ML algorithms to face the common constraints of memory and time [29]. To face that concern, classical distributed ML distributes the work among different processors, each performing part of the algorithm. Another current ML problem concerns the choice of ML algorithms and hyperparameters for a given task. For ML experts, this selection of algorithms and hyperparameters may use domain knowledge or heuristics, but it is not an easy task for non-ML-experts. AutoML was developed to combat this relevant issue [22]. The definition of AutoML can be described as the search for the best algorithm and hyperparameters for a given dataset with minimum human input. In recent years, a large number of AutoML tools was developed, such as AutoGluon [3], Auto-Keras [23], Auto-Sklearn [15], Auto-Weka [24], H2O AutoML [21], Rminer [10], TPOT [27], and TransmogrifAI [30]. Within our knowledge, few studies directly compare AutoML tools. Most studies compare one specific AutoML framework with state-of-the-art ML algorithms [15], do not present experimental tests [12,35], or are related to ML automation challenges [18–20]. Recently, some studies focused on experimental comparisons of AutoML tools. In 2019, [17] and [32] compare a set of AutoML tools using different datasets and ML tasks. In 2020, a benchmark was conducted using publicly available datasets from OpenML [33], comparing different types of AutoML tools, which were grouped by their capabilities [36]. None of the mentioned comparison studies considered the distributed ML capability for the AutoML tools. Furthermore, none of the studies used datasets from the domain of telecommunications risk management, such as churn prediction or fraud detection. 3 Proposed Architecture This paper is part of “Intelligent Risk Management for the Digital Age” (IRMDA), a R&D project developed by a leading Portuguese company in the area of software and analytics. The purpose of the project is to develop a ML system to assist the company telecommunications clients. Both scalability and automation are central requirements to the ML system since the company has many clients with diverse amounts of data (large or small) and that are typically nonML-experts. The ML technological architecture that is proposed by this work identifies and automates all typical tasks of a common supervised ML application, with minimum human input (only the dataset and the target column). Also, since the architecture was developed to work within a cluster with several processing nodes, the users can handle any size of datasets just by managing the number of cluster nodes. The architecture is illustrated in Fig. 1. 4 Lu´ıs Ferreira et al. Fig. 1. The proposed automated and scalable ML architecture (adapted from [14]). 3.1 Phases The proposed architecture assumes two main phases (Fig. 1): a training phase and a testing phase. Training Phase: The training phase includes the creation of a pipeline instance and the definition of its stages. The only human input needed by the user is the selection of the training dataset and the identification of the target column. Depending on the dataset columns, the each module defines a set of stages for the pipeline. Each stage either transforms data or also creates a model based on the training data that will be used on the test phase to transform the data. When all stages are defined, the pipeline is fitted to the training data, creating a pipeline model. Finally, the pipeline model is exported to a file. Testing Phase: The execution of the testing pipeline assumes the same transformations that were applied to the training data. To execute the testing pipeline the user only needs to specify the test data and a pipeline model (and a forecasting horizon in the case of time series forecasting task). The last stage of the testing pipeline is the application of the best model obtained during training, generating the predictions. Performance metrics are also computed and presented to the user. 3.2 Components The proposed architecture includes five main components: task detection, data preprocessing, feature selection, model training (with the usage of AutoML), and pipeline deployment. Machine Learning Task Detection: Set to detect the ML task of the pipeline (e.g., classification, regression, time series). This detection is made by analyzing the number of levels of the target column and the existence (or not) of a time column. A Scalable and Automated ML Framework 5 Data Preprocessing: Handles missing data, the encoding of categorical features, and the standardization of numerical features. The applied transformations depend on the data type of the columns, number of levels, and number of missing values. Feature Selection: Deletes features from the dataset that may decrease the predictive performance of the ML models, using filtering methods. Filtering methods are based on individual correlations between each feature and the target, removing several features that present the lowest correlations [4]. Model Training: Automatically trains and tunes a set of ML models using a set of constraints (e.g., time limit, memory usage). The component also identifies the best model to be used on the test phase. Pipeline Deployment: Manages the saving and loading of the pipelines to and from files. This module saves the pipeline that will be used on a test set, ensuring that the new data will pass through the same transformations as the training data. Also, the component stores the best model obtained during the training to make predictions, discarding all other ML models. 4 Materials and Methods 4.1 Experimental Evaluation For the experimental evaluation, we first examined the characteristics of the open-source AutoML tools. Then, we used the tools that could be implemented in our architecture to perform a benchmark study. In order to be considered for the experimental evaluation, the tools have to implement distributed ML. 4.2 AutoML Tools We first analyzed eight recent open-source AutoML tools, to verify their compliance with the project requirements. Auto-Gluon: AutoGluon is an open-source AutoML toolkit with a focus on Deep Learning. It is written in Python and runs on Linux operating system. AutoGluon is divided into four main modules: tabular data, image classification, object detection, and text classification [3]. In this article, only the tabular prediction functionalities are being considered. Auto-Keras: Auto-Keras is a Python library based on Keras [6] that implements AutoML methods with Deep Learning algorithms. The focus of AutoKeras is the automatic search for Deep Learning architectures and hyperparameters, usually named Neural Architecture Search [13]. 6 Lu´ıs Ferreira et al. Auto-Sklearn: Auto-Sklearn is an AutoML Python library based on ScikitLearn [28] that implements methods for automatic algorithm selection and hyperparameter tuning. Auto-Sklearn aims to free the user from the choice of an algorithm and the tuning of its hyperparameters using Bayesian optimization, meta-learning, and Ensemble Learning [16]. Auto-Weka: Auto-Weka is a module of WEKA, a ML tool that provides data preprocessing functions and ML algorithms that allow users to quickly compare ML models and create predictions using new data [34]. Auto-Weka aims to solve the Combined Algorithm Selection and Hyperparameter Optimization (CASH) problem, first established in [31]. H2O AutoML: H2O AutoML is one of the open-source modules of H2O, a ML analytics platform that uses in-memory data and implements a distributed and scalable architecture [7]. H2O AutoML uses H2O’s infrastructure to provide functions to automate algorithm selection and hyperparameter optimization [21]. Rminer: Rminer is a package for the R tool, intending to facilitate the use of Machine Learning algorithms. The focus of Rminer are the CRISP-DM phases of Modeling and Evaluation [8,9]. In the most recent version, Rminer uses more than 20 classification and regression algorithms. Also, since version 1.4.4, Rminer implements AutoML functions. TPOT: Tree-Based Pipeline Optimization Tool (TPOT) is an open-source AutoML written in Python. TPOT automates the phases of feature selection, feature engineering, algorithm selection, and hyperparameter tuning. It uses algorithms such as Decision Trees, Random Forest, and XGBoost, most of them from the Scikit-Learn library [25,27]. TransmogrifAI: TransmogrifAI is a tool that uses Apache Spark framework to automate ML applications. It is written in Scala and focused on the automation of several phases of the ML workflow, such as algorithm selection, feature selection, and feature engineering [30]. AutoML Tool Comparison: Table 1 presents the characteristics of the analyzed AutoML related to interface language, associated platforms, current version and if it contains a Graphical User Interface and distributed ML mode. For the experimental study, we selected H2O AutoML and TransmogrifAI, as these were the only tools from Table 1 that meet the distributed ML requirement. Table 2 presents the ML algorithms implemented by both tools. The last two rows are related to the stacking ensembles implemented by H2O AutoML: all, which combines all trained algorithms; and best, which only combines the best algorithm per family. A Scalable and Automated ML Framework 7 Table 1. Main characteristics of the analyzed AutoML tools (extended from [14]). Interface Language Associated Platforms Current Version Graphical User Interface Distributed Mode Auto-Gluon Python - 0.0.14 - - Auto-Keras Python - 1.0.9 - - Auto-Sklearn Python - 0.10.0 - - Auto-Weka Python RWEKA 2.6.1 DH2O AutoML Python R Scala AWS Azure Google Cloud Apache Spark 3.30.1.3 D D Rminer R - 1.4.6 - - TPOT Python - 0.11.5 - - TransmogrifAI Scala Apache Spark 0.7.0 - D Table 2. Algorithms implemented by H2O AutoML and TransmogrifAI (adapted from [14]). Algorithm H2O AutoML TransmogrifAI Decision Trees -D Deep Learning DExtremely Randomized Forest DGradient-Boosted Trees (GBT) -D Gradient Boosting Machine (GBM) DGeneralized Linear Model (GLM) DLinear Regression -D Linear Support Vector Machine -D Logistic Regression -D Naive Bayes -D Random Forest (RF) D D XGBoost D (only fully supported in Linux) - Stacking All (SA) DStacking Best (SB) D4.3 Data For the benchmark study, we used three real-world datasets from the domain of telecommunications, provided by the IRMDA project analytics company. The datasets are related to customer churn prediction (regression), event forecasting (univariate time series), and telecommunications fraud detection (binary classification). 8 Lu´ıs Ferreira et al. Churn Prediction: The churn dataset contains 189 rows and 21 attributes. The attributes of each row characterize a client and the probability for canceling the company’s analytics service (churn), as defined by the company. Table 3 describes each attribute of the churn dataset. Table 3. Description of the attributes of the churn dataset (adapted from [14]). Attribute Description tenure Time passed since the beginning of the contract streaming quality Contractualized display resolution ott video If OTT video is contractualized or not contract Duration of the contract payment method Contractualized method of payment product name Identification of the product platform Type of connectivity present in the contract financial status If the payment is late or regularized service latency Latency of the service dropped frames Number of dropped frames volume Information about volume duration Information about duration account number Account identification number service latency category Category of the service latency attribute dropped frames category Category of the dropped frames attribute volume category Category of the volume attribute duration category Category of the duration attribute tenure category Category of the tenure attribute account segment Age segment of the client equipment Equipment used by the client churn probability Probability of canceling the service (∈[0,1]) Event Forecasting: The event forecasting dataset contains 1,418 rows that correspond to records about telecommunication events of a certain type (e.g., phone calls). The events occurred from February to April of 2019, aggregated on an hourly basis, ranged from 3,747 to 56,320. The only attributes are the timestamp and the number of events in that interval, as described in Table 4. Table 4. Description of the attributes of the event forecasting dataset (adapted from [14]). Attribute Description Time Timestamp (format: yyyy-mm-dd hh:mm) datapoints Number of events A Scalable and Automated ML Framework 9 Fraud Detection: Each row of the fraud detection dataset contains the identification of A (sender) and B (receiver), and the classification of the phone call (“fraud” or “normal”). The dataset contains more than 1 million examples, which correspond to one day of phone calls from one of the company clients. The dataset attributes are described in Table 5. Table 5. Description of the attributes of the fraud dataset (adapted from [14]). Attribute Description A Identification of the call sender B Identification of the call receiver Result Classification of the call (“fraud” or “normal”) 5 Results 5.1 Experimental Setup The benchmark consisted of several computational experiments that used three real-world datasets to compare the selected AutoML tools (H2O AutoML and TransmogrifAI). The benchmark was executed on a machine with an i7-8700 Intel processor with 6 cores. Every AutoML experiment considered a holdout split that used 3/4 of the data as training set and 1/4 as test set. The split between train and test sets was random for two of the datasets (churn and fraud). For the event forecasting dataset, the division between train and test was ordered in time (since the data is ordered in time). Every AutoML execution implemented a 10-fold cross-validation during the training of the algorithms. Each AutoML tool optimizes a performance metric to select the best algorithms and tune the hyperparameters. We selected the Mean Absolute Error (MAE) for the regression tasks and Area Under Curve (AUC) for the classification data. Also, we computed additional metrics for the test data in order to further compare the tools. Additionally, we disabled time limits to allow the execution of all selected ML algorithms. From the algorithms presented in Table 2, we only disabled Deep Learning from the experiments, from H2O AutoML. First, because it required a greater computational effort, especially for the fraud detection dataset. Second, to achieve a more fair comparison with TransmogrifAI, since this tool does not include Deep Learning algorithms. For the churn dataset, the performance of the AutoML tools was measured with two scenarios. The first scenario (1) considered all the attributes of the dataset as input features for the ML algorithms. The second scenario (2) only uses a subset of the attributed as input features, derived from a previous feature selection phase. For TransmogrifAI the intention was to test the automatic feature selection characteristic. For H2O AutoML, the features of scenario 2 were 16 Lu´ıs Ferreira et al. 3. Auto-Gluon: AutoGluon: AutoML Toolkit for Deep Learning — AutoGluon Documentation 0.0.1 documentation (2020), https://autogluon.mxnet.io/ 4. Blum, A.L., Langley, P.: Selection of relevant features and examples in machine learning. Artificial Intelligence 97(1-2), 245–271 (dec 1997). https://doi.org/10.1016/s0004-3702(97)00063-5 5. Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002). https://doi.org/10.1613/jair.953 6. Chollet, F., et al.: Keras. https://keras.io (2015) 7. Cook, D.: Practical machine learning with H2O: powerful, scalable techniques for deep learning and AI. ” O’Reilly Media, Inc.” (2016) 8. Cortez, P.: Data mining with neural networks and support vector machines using the r/rminer tool. In: Perner, P. (ed.) Advances in Data Mining. Applications and Theoretical Aspects, 10th Industrial Conference, ICDM 2010, Berlin, Germany, July 12-14, 2010. Proceedings. Lecture Notes in Computer Science, vol. 6171, pp. 572–583. Springer (2010). https://doi.org/10.1007/978-3-642-14400-4 44 9. Cortez, P.: A tutorial on using the rminer r package for data mining tasks. Tech. rep., Universidade do Minho. Escola de Engenharia (EEng) (2015) 10. Cortez, P.: Package ’rminer’ (2020), https://cran.rproject.org/web/packages/rminer/rminer.pdf 11. Darwiche, A.: Human-level intelligence or animal-like abilities? Commun. ACM 61(10), 56–67 (2018). https://doi.org/10.1145/3271625, https://doi.org/10.1145/3271625 12. Elshawi, R., Maher, M., Sakr, S.: Automated machine learning: State-of-the-art and open challenges. arXiv preprint arXiv:1906.02287 (2019) 13. Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: A survey. arXiv preprint arXiv:1808.05377 (2018) 14. Ferreira., L., Pilastri., A., Martins., C., Santos., P., Cortez., P.: An automated and distributed machine learning framework for telecommunications risk management. In: Proceedings of the 12th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,. pp. 99–107. INSTICC, SciTePress (2020). https://doi.org/10.5220/0008952800990107 15. Feurer, M., Klein, A., Eggensperger, K., Springenberg, J.T., Blum, M., Hutter, F.: Efficient and robust automated machine learning. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 2962–2970 (2015), http://papers.nips.cc/paper/5872-efficient-androbust-automated-machine-learning 16. Feurer, M., Springenberg, J.T., Hutter, F.: Initializing bayesian hyperparameter optimization via meta-learning. In: Bonet, B., Koenig, S. (eds.) Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA. pp. 1128–1135. AAAI Press (2015), http://www.aaai.org/ocs/index.php/AAAI/AAAI15/paper/view/10029 17. Gijsbers, P., LeDell, E., Thomas, J., Poirier, S., Bischl, B., Vanschoren, J.: An open source automl benchmark. arXiv preprint arXiv:1907.00909 (2019) 18. Guyon, I., Bennett, K.P., Cawley, G.C., Escalante, H.J., Escalera, S., Ho, T.K., Maci`a, N., Ray, B., Saeed, M., Statnikov, A.R., Viegas, E.: Design of the 2015 chalearn automl challenge. In: 2015 International Joint Conference on Neural Networks, IJCNN 2015, Killarney, Ireland, July 12-17, 2015. pp. 1–8. IEEE (2015). https://doi.org/10.1109/IJCNN.2015.7280767 A Scalable and Automated ML Framework 17 19. Guyon, I., Chaabane, I., Escalante, H.J., Escalera, S., Jajetic, D., Lloyd, J.R., Maci`a, N., Ray, B., Romaszko, L., Sebag, M., Statnikov, A.R., Treguer, S., Viegas, E.: A brief review of the chalearn automl challenge: Any-time any-dataset learning without human intervention. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Proceedings of the 2016 Workshop on Automatic Machine Learning, AutoML 2016, co-located with 33rd International Conference on Machine Learning (ICML 2016), New York City, NY, USA, June 24, 2016. JMLR Workshop and Conference Proceedings, vol. 64, pp. 21–30. JMLR.org (2016) 20. Guyon, I., Sun-Hosoya, L., Boull´e, M., Escalante, H.J., Escalera, S., Liu, Z., Jajetic, D., Ray, B., Saeed, M., Sebag, M., Statnikov, A.R., Tu, W., Viegas, E.: Analysis of the automl challenge series 2015-2018. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning - Methods, Systems, Challenges, pp. 177– 219. The Springer Series on Challenges in Machine Learning, Springer (2019). https://doi.org/10.1007/978-3-030-05318-5 10 21. H2O.ai: H2O AutoML (June 2017), http://docs.h2o.ai/h2o/latest-stable/h2odocs/automl.html, h2O version 3.30.0.1 22. He, X., Zhao, K., Chu, X.: Automl: A survey of the state-of-the-art. arXiv preprint arXiv:1908.00709 (2019) 23. Jin, H., Song, Q., Hu, X.: Auto-keras: An efficient neural architecture search system. In: Teredesai, A., Kumar, V., Li, Y., Rosales, R., Terzi, E., Karypis, G. (eds.) Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019. pp. 1946–1956. ACM (2019). https://doi.org/10.1145/3292500.3330648 24. Kotthoff, L., Thornton, C., Hoos, H.H., Hutter, F., Leyton-Brown, K.: Autoweka 2.0: Automatic model selection and hyperparameter optimization in WEKA. The Journal of Machine Learning Research 18, 25:1–25:5 (2017), http://jmlr.org/papers/v18/16-261.html 25. Le, T.T., Fu, W., Moore, J.H.: Scaling tree-based automated machine learning to biomedical big data with a feature set selector. Bioinformatics 36(1), 250–256 (2020) 26. Oliveira, N., Cortez, P., Areal, N.: The impact of microblogging data for stock market prediction: Using twitter to predict returns, volatility, trading volume and survey sentiment indices. Expert Syst. Appl. 73, 125–144 (2017). https://doi.org/10.1016/j.eswa.2016.12.036 27. Olson, R.S., Urbanowicz, R.J., Andrews, P.C., Lavender, N.A., Kidd, L.C., Moore, J.H.: Applications of Evolutionary Computation: 19th European Conference, EvoApplications 2016, Porto, Portugal, March 30 – April 1, 2016, Proceedings, Part I, chap. Automating Biomedical Data Science Through Tree-Based Pipeline Optimization, pp. 123–137. Springer International Publishing (2016). https://doi.org/10.1007/978-3-319-31204-0 9, http://dx.doi.org/10.1007/978-3-319-31204-0 9 28. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., VanderPlas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in python. Journal of Machine Learning Research 12, 2825–2830 (2011), http://dl.acm.org/citation.cfm?id=2078195 29. Peteiro-Barral, D., Guijarro-Berdi˜nas, B.: A survey of methods for distributed machine learning. Progress in Artificial Intelligence 2(1), 1–11 (2013). https://doi.org/10.1007/s13748-012-0035-5 30. Salesforce: Transmogrifai (2019), https://docs.transmogrif.ai/en/stable/ 18 Lu´ıs Ferreira et al. 31. Thornton, C., Hutter, F., Hoos, H.H., Leyton-Brown, K.: Auto-weka: Combined selection and hyperparameter optimization of classification algorithms. In: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 847–855 (2013). https://doi.org/10.1145/2487575.2487629 32. Truong, A., Walters, A., Goodsitt, J., Hines, K., Bruss, B., Farivar, R.: Towards automated machine learning: Evaluation and comparison of automl approaches and tools. arXiv preprint arXiv:1908.05557 (2019) 33. Vanschoren, J., van Rijn, J.N., Bischl, B., Torgo, L.: Openml: networked science in machine learning. SIGKDD Explorations 15(2), 49–60 (2013). https://doi.org/10.1145/2641190.2641198 34. Witten, I.H., Frank, E., Hall, M.A., Pal, C.J.: Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann (2016) 35. Yao, Q., Wang, M., Chen, Y., Dai, W., Yi-Qi, H., Yu-Feng, L., Wei-Wei, T., Qiang, Y., Yang, Y.: Taking human out of learning applications: A survey on automated machine learning. arXiv preprint arXiv:1810.13306 (2018) 36. Z¨oller, M.A., Huber, M.F.: Benchmark and Survey of Automated Machine Learning Frameworks. Tech. rep., https://www.researchgate.net/publication/332750780