Automation of machine learning models benchmarking

Universidade do Minho Escola de Engenharia João Pedro Barros de Sá Automation of Machine Learning Models Benchmarking december, 2021 Universidade do Minho Escola de Engenharia João Pedro Barros de Sá Automation of Machine Learning Models Benchmarking Master Thesis Master in Informatics Engineering Work developed under the supervision of: Professor Doctor João Miguel Lobo Fernandes Professor Doctor André Leite Ferreira december, 2021 COPYRIGHT AND TERMS OF USE OF THIS WORK BY A THIRD PARTY This is academic work that can be used by third parties as long as internationally accepted rules and good practices regarding copyright and related rights are respected. Accordingly, this work may be used under the license provided below. If the user needs permission to make use of the work under conditions not provided for in the indicated licensing, they should contact the author through the RepositoriUM of Universidade do Minho. License granted to the users of this work Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International CC BY-NC-SA 4.0 https://creativecommons.org/licenses/by-nc-sa/4.0/deed.en ii Acknowledgements This words are dedicated to everyone involved in my academic journey so far, it’s success is in part thanks to you. To my parents, brother and girlfriend for all the support provided during this long year. It was thanks to all of you I got the strength necessary to overcome all the challenges, and finish another chapter in my academic and personal life. To Professor João Miguel Lobo Fernandes, thank you for providing all the support, guidance and mentorship over the year to achieve this document. To my work supervisor, André Ferreira, thank your for being my first professional leader and mentor, your guidance and confidence in my work was crucial towards the success of the work achieved throughout the year. To the MlOps team who together formed great moments of friendship and collaboration, it was a privilege to work with you and thank you for making this work possible in the end. To Pedro Vieira, for being a great friend and giving the most honest advice, always contributing with positive feedback leading to a higher quality standard in the work performed. To everyone at Bosch, thank you for providing a great work environment and contributing to this work. iii STATEMENT OF INTEGRITY I hereby declare having conducted this academic work with integrity. I confirm that I have not used plagiarism or any form of undue use of information or falsification of results along the process leading to its elaboration. I further declare that I have fully acknowledged the Code of Ethical Conduct of the Universidade do Minho. iv Resumo Automation of Machine Learning Models Benchmarking Na área de ciência de dados, o machine learning está-se a revelar uma ferramenta essencial para resolver problemas complexos. As empresas estão a investir em equipas de ciência de dados e Machine Learning para desenvolver modelos que apresentem valor para os clientes. No entanto, estes modelos são uma pequena percentagem de uma pipeline de projetos de Machine Learning (ML) e, para entregar um produto de ML completo, é necessário um número maior de componentes. DevOps é uma mentalidade de engenharia e um conjunto de práticas que visa unificar o processo de desenvolvimento e o processo de operações em um software, MLOps é um conceito similar a DevOps mas aplicado ao desenvolvimento e entrega de soluçoes de ML. O nível de automatização das etapas em uma pipeline de ML define a maturidade do processo de ML, que reflete a velocidade de treino de novos modelos com novos dados ou de treino de novos modelos com diferentes implementações. Um sistema de ML é um sistema de software, desenvolvimento e atualizações continuas são necessárias para garantir um sistema que escale conforme as necessidades. O principal objetivo desta tese é apoiar a criação de um sistema integrado de ML com uma arquitetura que proporcione a capacidade de ser continuamente operada em um ambiente de produção. Um conceito para avaliação de desempenho de algoritmos deve ser elaborado e implementado. O principal objetivo é melhorar e acelerar o ciclo de desenvolvimento de modelos de ML na empresa. Para atingir este objetivo surge a necessidade de definir uma arquitetura com especificações e a implementação de processos automatizadas num pipeline de ML existente, este processo têm como objetivo alcançar uma ferramenta de benchmark de modelos, com capacidade de analisar o desempenho do modelo, um motor de inferência e um banco de dados para armazenar todas as métricas computadas. Um sistema baseado em IA em desenvolvimento fornece o caso de estudo para desenvolver e validar a arquitectura. Os avanços atuais na área da condução semiautomática introduz a necessidade de sistemas de monitoramento que podem localizar e detectar eventos específicos no veículo. Os conjuntos de sensores são instalados dentro da cabine para alimentar sistemas inteligentes que visam analisar e sinalizar certos comportamentos que podem impactar a segurança e o conforto dos passageiros. Palavras-chave: Engenharia Software, Aprendizagem Máquina, Ciência dados, DevOps, MlOps … v Abstract Automation of Machine Learning Models Benchmarking In the field of data science, ML is proving to be a core feature to solve complex real-world problems. Businesses are investing in data science and ML teams to develop AI based models that can deliver business value to their users. However, these models are only a small fraction of an ML project pipeline, and to deliver an end to end ML product, a greater number of components are needed. DevOps is an engineering mindset and a set of practices that aims to unify the development process and the operation process on software. MlOps is a similar concept to DevOps but applicable to the development and delivery of ML based solutions. The automation of the steps in a ML pipeline defines the maturity of the ML process, reflecting the velocity of training new models given new data or training new models given new implementations. An ML system is a software system that can support development, provide continuous integration and continuous delivery apply to help guarantee that one can reliably build and operate ML systems at scale. The main objective of this thesis are to support the creation of an integrated ML system with an architecture that provides the ability to be continuously operated in a production-like environment. Furthermore, a concept to evaluate the performance of algorithms shall be devised and implemented. The end goal is to improve and accelerate the ML development lifecycle. To achieve this goal surges the need to define an architecture alongside specifications and the implementation of several automated steps into an existing ML pipeline. To improve and accelerate model development an model engine benchmark tool is devised capable of several features, including the ability to have dashboards for model performance evaluation, an automatic inference engine, performance metrics for the model and a database to store all the computed metrics and metadata. An AI-based system under development provides the case study to develop and validate this architecture. The current advances of semi-automated driving introduce the need for monitoring systems to scan and detect specific events in the vehicle. Sensor clusters are installed inside the vehicle cabin to feed data to intelligent systems that aim to analyze and red flag certain behaviours that can potentially impact passengers safety and comfort while using the vehicle. Keywords: Machine Learning, Software, MlOps, DevOps, Data Science, Pipelines, Automation vi Contents List of Figures ix Acronyms xi 1 Introduction 1 1.1 Contextandmotivation .............................. 1 1.2 ChallengeatBOSCH ............................... 2 1.3 Thesisgoals ................................... 2 1.4 Documentstructure................................ 3 2 State of the art 5 2.1 Software development methodologies . . . . . . . . . . . . . . . . . . . . . . . 5 2.1.1 The waterfall approach . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1.2 Theagileapproach............................ 6 2.2 DevOps...................................... 6 2.2.1 Technicaldebt.............................. 7 2.2.2 DevOpsCulture ............................. 7 2.2.3 DevOpstechniques............................ 8 2.3 Development in machine learning . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3.1 Crisp-Dm ................................ 9 2.3.2 Hidden technical debt in Machine Learning . . . . . . . . . . . . . . . . 10 2.4 MLOps...................................... 11 2.5 Best practices for model deployment . . . . . . . . . . . . . . . . . . . . . . . 14 2.5.1 The problem of concept drift . . . . . . . . . . . . . . . . . . . . . . . 14 2.5.2 MlOps frameworks for the data engine . . . . . . . . . . . . . . . . . . 14 2.5.3 MlOps frameworks for model benchmark engine . . . . . . . . . . . . . 15 2.5.4 Model benchmarking methodologies . . . . . . . . . . . . . . . . . . . 15 2.6 Model explainability methodologies . . . . . . . . . . . . . . . . . . . . . . . . 16 2.6.1 Lime values explainability . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.6.2 Shapvalues ............................... 18 vii 1.4. DOCUMENT STRUCTURE Figure 2: Model engine benchmark components First, the Model performance dashboard is a tool to simplify the process of evaluating an ML model. The dashboard will contain key performance indicators of the model, thus facilitating the comparison process between models. The Inference engine is a component of the ML system pipeline that applies logical rules to a knowledge graph to derive new information. Inference engines have gained much traction due to the current machine learning boom and are extremely useful at providing new insight and knowledge. Lastly, in the current machine learning panorama, many models algorithms are built, trained, and evaluated, all with different architectures, data pipelines, and parameters, generating an issue. The Model performance database is a component designed to automate and facilitate model storage. It stores all the information related to each model, including key performance indicators and data pipelines used. It also allows for a versioning system to know exactly everything about each model and its evaluation. This solution aims to automate, simplify and speed the whole process of building and deploying models in the ML systems developed. 1.4 Document structure The remaining document structure is the following: •State of the art - This section targets a complete review of the literature around the topic of 3 CHAPTER 1. INTRODUCTION this thesis, including problems in software development methodologies and the solutions that were proposed over the years to solve them. Machine learning systems can be software systems, so the same challenges can also exist in the current ML system development. A complete review of these topics will be presented in this section. •Early work - This chapter shows the early work towards the development of the MlOps automation pipeline. •Objectives and results - This chapter outlines the requirements defined for the solution. •Design and architecture - Full explanation of the implementation process to achieve the architecture defined in the previous steps. All the decisions made, implementation choices, and outcomes. •Conclusion - This chapter provides the experiment setup explanation and the entire outcome of those setups. It concludes the work achieved so far. 4 Chapter 2 State of the art This chapter provides an in-depth literature review on machine learning and software systems development approaches. What was used in the past, what is used now, and the main advantages and problems that arose from each. Since machine learning is a sub-field of Artificial Intelligence (AI), some software development problems extrapolated from one to the other, so before we tackle the approaches currently used in Machine Learning (ML), a brief look at the past is needed to further enhance the perception of the current state of the art in development methodologies. 2.1 Software development methodologies This sub-chapter provides the foundation on how the current state of the art came to be. It reviews some of the technical difficulties and the solutions proposed in the past to understand the present. 2.1.1 The waterfall approach Before the 21st century, the waterfall approach was a common methodology used to deliver software inside an organization. This approach was divided into multiple phases, as described in figure 3. Each phase had to be completed before proceeding to the following stage, and this meant that changes required after the initial requirements phase were complex and costly to implement, and the process was very rigid, maintenance in the software was also a difficult task, (Van Casteren, 2017). At the beginning of the century, software was in more demand than ever, and the waterfall approach started to show limitations dealing with the volatility in software requirements, (Rahman, 2019). The development cycle was long, rigid, and could no longer keep up with the increasing demand for software supply and maintenance processes. 5 CHAPTER 2. STATE OF THE ART Figure 3: Waterfall development approach, by (Van Casteren, 2017 ). 2.1.2 The agile approach In 2001, due to the limitation of the waterfall approach, a group of developers discussed a new methodology for software development, which led to the creation of the agile manifesto, (Martin Fowler, 2001). This manifesto introduced a set of rules and methods to improve interaction and communication within teams. Agile approaches required face-to-face interaction and short iterative cycles with dynamic feature planning, (Van Casteren, 2017). The agile approach was meant to make software delivery fast and continuous. It intends to integrate business people with developers, facilitate late requirements change, motivate individuals, improve the overall team performance, and schedule regular meetings to discuss how to become more effective. Simplicity in design choices was welcomed, (Kumar and Bhatia, 2012). Multiple papers surged over the years on tackling agile development, being scrum one of the most popular. It named the development iterations cycles as sprints, which usually lasts between 1-4 weeks. Team meetings called daily scrums happen every day and serve to discuss what has been done, difficulties, and future work, but the original manifesto’s core philosophy remained unchanged. 2.2 DevOps DevOps is a set of practices that combines software development (Dev) and IT operations (Ops) with one key difference with respect to the agile approach, which while it was great in simplifying the development process between business people and developers, the operations process was left out. To solve this limitation, DevOps was born, proposing methods to approximate and combine the development and operations process into one. 6 2.2. DEVOPS 2.2.1 Technical debt Technical debt is a term coined by Ward Cunningham in 1992. The term refers to a metaphor between debt in the financial system and the debt in software development, (Cunningham, 1992). This debt refers not necessarily to lousy coding. Most of the time, it represents a trade-off between good practices and simplified implementations to save time and speed up the delivery process of new features. As the development process matures over time, testing teams report more and more bugs resulting from these cheap implementations creating a deficit over time that needs to be paid. This phenomenon can create a problem where software maintenance is compromised because as each situation is dealt with, more issues arise in the process. With the introduction of agile methodologies, dealing with technical debt became more straightforward, but there still was little connection between developers and operators. DevOps surged as a way to mitigate the problem of technical debt, streamlining the whole development process, from software requirements to operations. While this helped manage technical debt, it has not eliminated nor was it expected to eliminate the problem. 2.2.2 DevOps Culture Gene Kim, one of the biggest contributors to the topic states in his book, The DevOps Handbook that three principles are sustaining a good DevOps practice and implementation, (Kim, Humble, Debois, and Willis, 2016). The first principle is about enabling left-to-right workflow and communications from the development to operations and costumers, removing the barriers between different teams or departments and promoting cooperation. The work is made visible and accessible to maximize flow, the updates are minor but frequent, and quality is preferable over quantity. This leads to a smaller time requirement to fulfil requirements, all while increasing quality. The result of this practice is continuous build, continuous integration,continuous test and continuous deployment. The second principle is all about enabling fast feedback flow from right-to-left from the operations and costumers to developers in all phases of the work. Adopting this principle allows for much quicker problem detection and consequent fix, permitting problems to be solved at the core, creating quality in the product, and reducing fatal failures to occur down the line. This principle is to contrast with the first, see figure 4. The third principle is all about building the DevOps culture and mindset, to ensure a high trust mentality that supports a dynamic, disciplined environment. The scientific approach to building products with tolerance to risks, making knowledge gained from failure and success alike, to complement this the shorter feedback loops allow for safer systems which increase the tolerance for taking risks and learning with them. A team adopting these principles makes them evolve faster than the competition. 7 CHAPTER 2. STATE OF THE ART Figure 4: DevOps communication and feedback flow 2.2.3 DevOps techniques To adopt DevOps there are several techniques to apply, Manish Virmani stated numerous techniques, (Virmani, 2015), as follows: •Continuous planning states that business plans need to be agile to meet up with fast market changes. Frequent interim checkpoints to overview the current state and adjust as needed. Without DevOps, it’s hard for developers to keep up with change, but with DevOps, the increase in communication and feedback between developers and customers allows for a better overhaul flow. •Continuous integration or Continuous Integration (CI) for short is a process where a team automates most of the tasks between development and operations. It involves teams integrating their work regularly and automating tasks like building, testing, and validating. In the end, this achieves faster bug finding and fixing and a greater standard of quality in the software with lower costs. This is what makes CI a stone pillar of a good DevOps implementation. •Continuous deployment is the act of getting all types of updates into the hands of users as fast and safely as possible, without worrying about what type of update it is, whether a bug fix, a new feature or even a large scale update. This reduces friction points inherent in the more traditional deployment or release processes. In other words, the application is always in a production-ready state. •Continuous testing is all about automating every step possible when testing. If an action is repetitive over time, it must be automated. Currently, there is a large number of technologies in the market that simplify this process. 8 2.3. DEVELOPMENT IN MACHINE LEARNING •Continuous monitoring, due to testing better and early, there is an opportunity to observe critical parameters and react to unseen events faster. 2.3 Development in machine learning Some machine learning applications share a set of similar characteristics with more traditional software applications. In many cases, they are almost identical, but with an extra ML model layer, (“MLOps”, 2020). Due to this, much of the same conventional problems are present, with the ML system introducing new concerns that need to be addressed. This section provides a complete analysis of ML development methodologies state of the art and the work being done to improve ML system pipelines. In figure 5 a typical ML system structure is presented. Figure 5: Components in a ML system, (“MLOps”, 2020 ) 2.3.1 Crisp-Dm With the rise of machine learning, one of the methodologies that gained the most popularity is CrispDm , short for Cross Industry Standard Process for Data Mining, This methodology is an industry standard consisting of six phases that describe most machine learning pipelines, see 6. Each step of the pipeline described is an important phase and are explained as follows: •Business understanding is a crucial step. It consists of understanding the underlying business that we want to apply machine learning, what customers want to accomplish and what data can lead us there. If we start working on a problem without previous knowledge, it won’t be easy to know which data to collect and the best way to treat it. •Data understanding is the step where we gain in-depth information about the problem. It usually starts with an extensive exploration of the data with statistic analysis and data visualization. 9 CHAPTER 2. STATE OF THE ART Figure 6: Crisp-DM pipeline by Hassanien, 2019 •Data preparation is generally one of the most time-consuming parts of the pipeline. It’s infrequent to obtain the data ready to fit the algorithms. Most of the time, we have raw collected data that is unsuited for our goals. Generally, we want each record in a row and each attribute in a column, but this is a general rule of thumb as video or audio data have different needs. In this phase of the pipeline, we already have information on the quality of the data, for instance, how many missing values, if the dataset is balanced, or simply if there is enough data for an ML algorithm to learn. •Modelling is about creating the model. First, there is the need to choose the technique that best suits our problem and data, then a test design is generated that may involve splitting the dataset into training, validation, and test subsets. The next step is to build the model and fine-tune it to get the best results possible. In this phase, we can go back to the data preparation phase and make changes to best suit our current model. •Evaluation is the phase where the model is evaluated, in this step we check if the model performs well and answers to the business needs, then an evaluation of the entire pipeline is done. In this stage a decision is made whether to deploy the model or return to the first phase to start again and change the approach. •Deployment is the last stage of the pipeline since a model isn’t useful if it cannot be accessed by its customers. In this stage, a deployment plan is built and a monitoring system is created to help future maintenance in the model. 2.3.2 Hidden technical debt in Machine Learning In chapter 2.2.1 the term technical debt is introduced, following that definition, (Sculley et al., 2015) introduced in a published article in 2015 the preeminence of ML systems to develop a very specific type of 10 2.4. MLOPS technical debt, which he called Hidden technical debt stating that ML systems have all the traditional problems of traditional coding plus specific problems that arise from the ML components. Furthermore, in traditional software, technical debt is code related, but in ML applications, Hidden technical debt is code and data-dependent, since an ML model learns from data, any change to the model or it’s data changes everything, (Alahdab and Çalıklı, 2019, Sculley et al., 2015 and Sculley et al., 2014) exemplifies this phenomenon in an article studying and finding patterns of hidden technical debt in ML systems. 2.4 MLOps DevOps revolutionized traditional software development. ML system development is still in an infant state compared to software, but the area has been receiving much more focus in recent years. In the academic community, most of the attention goes to the early stages of the ML pipeline, preparing and exploring data and model development, but in the real world, model deployment and maintenance are as important if we want to give use to our ML systems, so automating our system pipeline becomes a crucial step to have reliable results in real-life applications, (Arnold et al., 2020). MlOps came to solve many of the issues of ML development by applying DevOps practices in ML systems. According to ((“MLOps”, 2020)), ML systems differ from traditional software and fail in different areas. The steps in a traditional data science workflow are as follows: •Data extraction is at the beginning of every data science workflow, since ML isn’t possible without data. In this phase, data is collected from one or multiple relevant sources. •Data analysis corresponds to the phase where the raw data collected in the previous phase is analyzed. Once this process is completed, the team gets familiar with the data available, its characteristics, and the work that will need to be performed in the next phase. •Data Preparation is one of the steps that are generally more time-consuming. It involves cleaning the raw data, treat missing values and perform feature engineering. In the end of this step we have the data ready to feed our model. •Model training is where the team implements and tests with different algorithms and different hyper-parameters tune. This outputs the best-trained ML model. •Model evaluation, the model is validated for predictive value, and if it passes, it’s ready to be deployed in a real-world scenario. •Model serving is when the model is deployed in the real world in a specific environment, this can be a web API, mobile or desktop application, among other possibilities. 11 CHAPTER 2. STATE OF THE ART •Model monitoring is the final step of a usual Data science workflow, the model is monitored, and if key performance indexes start dropping, maintenance is required, and an ML task in this process is performed. MlOps is all about automating every single step that is repetitive along multiple ML pipelines, and the level of automation defines the maturity of the MlOps implementation. A bigger maturity means a faster end to end process. An entry-level ML system has no automation in the pipeline. This is very common in academic projects and companies starting their first ML projects. The typical pipeline in this type of project doesn’t include continuous delivery or integration, no autonomous monitoring, see figure 7. This type of system usually doesn’t age well due to all the limitations that will contribute to hidden technical debt. Figure 7: First level of maturity, by (“MLOps”, 2020 ) In the next level, we find a type of ML systems that have a more mature MLOps implementation. These types of systems have a significant level of automation implemented. This includes the data ingestion pipeline to feed the model that ensures continuous training, finally, the consequent continuous serving of the model to the final customer. At this level, we can train new models given new data very fast. However, the process is still slow and not ideal for making changes in the ML algorithm, see figure 8. The final level of maturity is the ideal MLOps ML system with fully-fledged CI/CD, (continuous integration / continuous delivery), implementations in this stage. The continuous integration usually has features like automatic testing for model performance, convergence and behavior checks between different pipeline components. The continuous delivery implies very fast delivery of changes in the pipeline to a production environment, this level of maturity is expected to have a very fast implementation given new data or given a new algorithm, see figure 9. One of the key parts of every MLOps implementation is to improve the system’s quality in the long run, as ML systems don’t age in the same way as traditional software systems. To combat this phenomenon, 12 2.8. SUMMARY The Hellinger distance measures the similarity between two distributions. The formula to calculate the Hellinger distance is in figure 12, the formula returns values closer to zero if both distributions are very similar and closer to one when they are very different from each other. Figure 12: Hellinger distance The average pairwise distance iterates over all observation in both distribution and returns the average computed distance, the formula is in figure 13, euclidean distance is the distance formula in this case, but any formula that can predict the distance between observation work with the average pairwise formula. Figure 13: Average Pairwise distance In the literature, (Breck, Cai, Nielsen, Salib, and Sculley, 2017) explores interesting approaches like data statistical analysis and data schemes that include the normal range expected for the feature, this way if data starts deviating from the normal standard at the time the model was trained, a warning can be provided to the developer who can act and avoid malfunction in the system due to data drifting, this process allows scoring and quantify changes in data. However, metrics analysis needs to be observed as well, which present challenges. Furthermore, there is no way to compare the predictions with the ground truth as the data is not labelled in real-time and requires human intervention, so this presents challenges that need to be solved. Data drift is a normal phenomenon that will always present itself over time, like for example consuming habits changing year after year. However, certain events can render an ML algorithm useless overnight, for example, a global pandemic that forced the entire population to use a face mask, making a huge shift towards online sales and led the global markets to an astonishing volatility rate, this led to huge implications in facial recognition applications, recommendation algorithms underperforming and stock market algorithms making huge mistakes. 2.8 Summary In conclusion, developing an end to end ML application has many factors to consider. Adopting the best MlOps practices means a faster development cycle and post-deployment updates, the gains obtained with MlOps depend on the maturity of the implementation. To assist with the automation of the pipeline, many frameworks are present in the market that help the process. Each has its main advantages and disadvantages for each use case. 19 CHAPTER 2. STATE OF THE ART Choosing the best algorithm is complex, and good benchmark methodologies in the market allow for comparisons between different models. Some methodologies consider inference cost, training cost, or both, and depending on the use case all have their advantages and disadvantages. Even the best algorithm won’t stand the test of time as data drift can and will impact the model’s performance. Many monitoring methodologies are being studied to combat this phenomenon, which applies techniques to control and signal potential data drift. 20 C h a p t e r 3 Early work A car has unique particularities for collecting data due to its fast pace movement, resulting in potential significant changes in the air quality and properties for the duration of the trip. This early work subsection intends on prototyping a solution for collecting vast amounts of data from real-world circulating vehicles, and study their properties to find value in them. This value can be to detect dangerous levels of smoke or distinct noise that can indicate danger to the passenger or physical harm to the vehicle. Before the data is ready to be fed into Ml algorithms, many steps are needed, like collecting it, treating it, and ensuring reliability in the data. This process is aimed to be automated, so a pipeline among the algorithms and the data source is constructed. After the model is trained, a pipeline between the training and deployment will also be constructed to streamline and facilitate deployment, benchmarking, and monitoring new models. This automation will lead to a much faster end to end cycle and provide the ability to update and maintain the Ml application easily. 3.1 Data collection - Hardware solution 3.1.1 Sensors and computational device To collect data from the vehicle, a computational device and sensors are needed. Sensors collect raw data from the environment, decode the data, and record it to an appropriate storage solution. The following hardware was selected •Pm2.5 Sensor is a particle sensor capable of detecting multiple sizes of particles in the air. •Uma8 microphone array is a high definition 8 channel microphone array that that collects audio from the environment. 21 CHAPTER 3. EARLY WORK •Bme680 is a gas sensor capable of measuring a wide range of gases, pressure, humidity, and temperature. •Raspberry pi4 model B is the computation device that manages the sensors and collect the data from them in a database These sensors were mainly selected because they can capture the data needed and were available in the company stock of sensors. The computational device is chosen because it offers the computational power required and can run a Linux distribution that runs the software necessary to manage the sensors. 3.1.2 Casing solution The results obtained in a given test vehicle needs to be reproducible and generalized for all vehicles. The sensors must be installed in identical locations with identical configurations to avoid different noise between different setups and to minimize noise from the environment. This requires a generalized setup. To solve the installation problem, a casing design was developed using a 3d Printer. This design needs to allow for the correct accommodation of all the sensors. For instance, the microphone cannot have sound blocked, and proper airflow needs to be supplied to the gas and particle sensor; otherwise, they will not function correctly. Furthermore, the installation location is in the car windshield, so it needs to be attached without scratching or damaging the glass when installed. With all the requirements addressed, the design process can start, a free CAD software was used, and a two-level design was conceptualized. This allows for airflow and sound to pass to the sensors while not overcomplicating the manufacturing(printing) process. The results can be seen in figure 14 Figure 14: Design’s levels design This design is printed and the final result is displayed in figure 15. The design provides a simple, effective, and pleasant casing solution that won’t distract the driver and can blend in with the natural aesthetics of the car. 22 3.2. SOFTWARE SOLUTION Figure 15: Printed multi level design 3.2 Software solution To manage and coordinate all sensors, Robot operating system (Ros) is used. ROS allows us to treat each sensor as a node that can communicate with a central node, usually called the listener that subscribes to channels and receives messages from the sensor nodes publishing in those channels. This approach allows to achieve true sync between data from different sources, a time synchronizer filter is used to eliminate incoherent data that arrives in different time sources, the data is then stored in a file structure called bagfile that allows storing raw data with different formats which are highly beneficial since each sensor has a unique data schema, this architecture can be visualized in figure 17. Figure 16: Component diagram for data collection tool This application uses a launch script to initiate all nodes at the same time. The node sensor registers their channels in the ROS API, and the listener node subscribes to them and starts waiting for messages. The sensors start collecting data and are timed in predefined intervals to publish the data in the specific data schema. The listener hears the messages arriving in different channels and verifies the time synch between them to check if consistency is maintained; if everything is good, they are stored in a database or otherwise deleted. This flow can be observed in 17 23 CHAPTER 3. EARLY WORK Figure 17: Activity diagram for data collection tool 3.2.1 Data visualization tool The data collection tool is created and has a standardized output meaning the data treatment can be automatized, for this reason, the data visualization tool has requirements specified to allow the automation of data between the database and the visualization tool. To achieve this the tool must interpret and decode the bagfiles that are the output of the collection tool, then extract and treat the data by interpreting how many sensors are registered and give the user the choice to select the ones that have interest for the current analysis, then show the graphs and statistical information from the data, see figure 18. Figure 18: Component diagram for data visualization tool This four components in figure 18 work as follows: 24 3.2. SOFTWARE SOLUTION •DataLoaderService is a collection of functions that loads raw data and executes the necessary processes to make it usable •Data is the component that represents the instance of data, providing functions and information on it’s domain •DataController is a comunication layer between the model class Data and the UserInterface, this apprach intents on centralizing all the requests. •UserInterface is the visual interface of the software and makes interaction with the tool simplified. This application is designed to be used through the user interface, all the functions are explained in the activity diagram in figure 19 Figure 19: Activity diagram for visualization tool 25 Chapter 4 Objectives and results 4.1 Problem definition The development of an end to end Machine Learning (ML) pipeline is a very complex process. A concept to evaluate the performance of algorithms shall be devised and implemented to support the choice of the best model and an inference engine that provide feedback for the predictions and assure model reliability. The end goal of the thesis is to accelerate and improve the company already existing ML development pipeline by automating and improving the existing steps. The research found that the best model is a highly subjective matter, with the use case has significant implications on benchmark metrics. In this section, a few considerations are made to achieve the final goal. 4.1.1 Benchmark consideration The ultimate problem for this project is to make impact detection and classification, this means the model receives data from the sensors and predicts if an impact happened and what type of impact to understand damages done to the car, based on this goal a better decision for benchmark metrics can be made. In many ML projects inference times are an essential subject, however, the goal of the current ML implementations is to detect and classify damage impacts on a car exterior, since we have no intention on predicting upcoming events, inference times are not important. This means that the correct metrics are the single most important factor in evaluating models. Different models with different role all flow from the pipeline and arrive at the benchmarking engine, each must take into consideration different evaluation metrics to correctly assess its usefulness and performance. The data scientist must be the responsible actor who chooses the correct metrics to evaluate the model. 26 4.2. OBJECTIVES The benchmark engine platform’s role is to have a standard that allows for the benchmark of every metric and to display the relevant information on the dashboard. The web dashboard has the role to intuitively display model key performance indexes and showcase the best models for each model type present in the system. 4.1.2 Concept drift in the context Data drift is certain to occur in every data-related problem, but it’s essential to consider how it can affect the performance in this specific use case. In this context, data drift can occur due to several reasons, some can be easily identifiable and some not. For instance, if a sensor starts to misbehave, it can be easily identified with a monitoring tool. However, suppose the car manufacturing process or the materials it’s made changes over the years to make it safer. In that case, the data to a similar incident can change drastically, like the sound of the collision or less g-force registering, making the model obsolete. A correct monitoring approach must be deployed to detect the occurrence of data drift in the data received by the model, this can be done for instance by applying the Hellinger distance, figure 12. If the same type of collision data starts differentiating a lot from training to real-world, then this formula will start to output values closer to 1. A case study needs to be performed to discover the best metric to measure such deviations or if such metrics can be helpful in the dataset used. This system is also responsible for keeping benchmarking each model has new data that corresponds to the original query is available. This process helps detect model performance over time and ensure the model is still performing as intended. 4.1.3 Continuous deployment and continuous training Ensuring a pipeline capable of continuous deployment and training is essential. Due to the enterprise nature of this application, downtime must be reduced to the minimum, so it’s important to design a pipeline that can perform updates in real-time to the model. So continuous training ensures concept drift doesn’t make the model outdated, and continuous deployment ensures the model is updated. The benchmark engine integrated into the pipeline helps choose the best model in the current setting. 4.2 Objectives The end goal is to design and build a pipeline capable of improving and accelerating the current ML development cycle in the organization, this implies to facilitate, improve and automate most of the steps and tasks already existing. In the end, time and human costs must be significantly reduced. This thesis end goal is to provide a solution that can tackle the final steps of the pipeline. After the model is trained, the solution must automate every action necessary and display the final key performance indexes and comparisons between different models in the web dashboard. The model benchmarking 27 CHAPTER 4. OBJECTIVES AND RESULTS engine needs to work and communicate with the previous components from the pipeline and respect the standard defined by the team. To tackle the problem, three-component must be devised. These components are as follows: • Model benchmark engine, capable of indexing, versioning and benchmarking each model that arrives while also performing continuous evaluations on existing models. • Web engine, capable of displaying all information about each model and benchmark while also providing a comparison tool • Database solution, capable of storing and managing all the information required by the system. 4.3 Requirements To solve the problem defined above, a combination of three components will be developed, each to solve a set of challenges related to the end stage of the pipeline. Combined, they form the model benchmark engine referenced in this document. These tools address the general complication in ML development, from model training to deployment and monitoring. The following requirements are defined for the system. 4.3.1 Functional requirements The system must allow to: •Receive new models from the pipeline. Models will flow from the model training engine automatically, a standard is necessary to send and receive models for communication and usability purposes. •Probe if new Data is available. A model comes associated to a dataset using its embedded metadata and a dataset Tag, new data relevant to the model may be available with time making it necessary to index and process it for the appropriate models. •Trigger benchmark action automatically for new models or new data. The first benchmark needs to be triggered when possible by the system, however this doesn’t mean the end of the model evaluation process, each model is set to be continuously benchmarked as new data is made available. •Store all the relevant metadata about each model / benchmark / dataset. The end user must have access to all relevant model information, this information is processed and stored in the database for further utilization. 28 5.2. MODEL BENCHMARK ENGINE DESIGN AND ARCHITECTURE OVERVIEW 5.2.1 System use cases Considering the problem defined, the user of the model benchmarking engine, a data scientist, does not have any interaction with the underlying system and only consults the information outputted and provided by the web dashboard. The user has the following use cases: •UC1: The user must be able to manually insert or delete models from the database. •UC2: The user must be able to access the database and perform CRUD operations. •UC3: The user must be able to filter each information table in the dashboard. •UC4: The user must be able to order each information table by column. •UC5: The user must be able to compare different model benchmarks. •UC6: The user must be able to have results filtered by dataset. •UC7: The user must be able to see all the metadata collected about the model. •UC8: The user must be able to access a model explainability dashboard. •UC9: The user must be able to access dataset explainability dashboard. •UC10: The user must be able to access all logging information in the system. •UC11: The user must be able to download individual models. Most of these use cases require that the automated underlying function work as expected. The system is responsible for receiving new models, storing the metadata and dataset information in the database, performing benchmark operations, and storing the metrics. Finally, it also runs operations to compute SHAP values and charts. The main actor of the system is the data scientist, which is denominated as the main user of the system. Most features and use cases implemented have the purpose to improve the data scientist ability to gather knowledge about each model and the decision-making process. Each Use Case (UC) is explained in the following use case diagrams. For example, in figure 23 the UC1 is explained, the model is inserted using the web dashboard which uploads the file to an object store solution, an internal scheduled process fetches for new models in a specified interval and if it founds one makes a request for the file, it then parses the embedded metadata and stores it in the database respecting the data schema defined. The sequence diagram in figure 24 showcases manual CRUD operations to the database, UC2 that may be necessary to perform. The web dashboard provides the interface for these operations, and when confirmed, the changes are executed in the database engine, making the process as user-friendly as possible for the user. All data tables in the web dashboard must be dynamic. This means each column must have the possibility to order by ascending or descending and a custom search bar to filter the content in each table. UC3, UC4 is described in the sequence diagram in figure figure 25. Another essential use case is UC5 and UC6. A tool must be devised that allows comparison between benchmarks of the same model or different models, and it is also important to filter all results by dataset to 35 CHAPTER 5. DESIGN AND ARCHITECTURE Figure 23: UC1: Sequence diagram for manual model insertion Figure 24: UC2: Sequence diagram for manual CRUD operations Figure 25: UC3, UC4: Sequence diagram for data filtering 36 5.2. MODEL BENCHMARK ENGINE DESIGN AND ARCHITECTURE OVERVIEW find the best performing model for a given dataset or the best performing model for a project use case like a damage detector or damage locator, the sequence diagram in figure 26 explains the intended sequence for this process. Figure 26: UC5, UC6: Sequence diagram for benchmark and results filtering UC7 describes the process of a user getting all the information available about a model. This includes all its metadata like the name, timestamp introduced, and all the benchmarks information performed to the model like the metrics results, datasets used, distribution matrices and more. This information can be consulted by requesting the model page, and the process is described in the diagram in figure 27 UC8 is about the user requesting and accessing a model explainability page. Explainability refers to the model hyperparameters and tuning information and the SHAP explainability charts explored in state of the art. Each model should have SHAP explainability. The process of displaying this information is in figure 28. This information should already be computed after the benchmark is performed and the data stored in the database. UC9 UC9 is a feature requested so the end-user can understand what is inside a specific dataset used for a benchmark. Although the process is explained in the sequence diagram of figure 29, this is different from other information pages, all data statistics are computed outside the benchmarking engine component and instead computed on the data ingestion engine, due to interoperability requirements between the two, a standard must exist to guarantee the correct dataset is requested. UC10 is a use case requirement because the end-user will only access the web dashboard. If an error happens in the background components, it is important to have access to the logs. Logs include system failures, exceptions and also critical performed operations like benchmarking being executed on a model. 37 CHAPTER 5. DESIGN AND ARCHITECTURE Figure 27: UC7: Sequence diagram for ordering a model page. Figure 28: UC8: Sequence diagram for ordering an explainability page. 38 5.2. MODEL BENCHMARK ENGINE DESIGN AND ARCHITECTURE OVERVIEW Figure 29: UC9: Sequence diagram for ordering a dataset explainability page. The sequence diagram in figure 30 explains the log collection and storing process, and the dashboard provides the information. Figure 30: UC10: Sequence diagram for obtaining logging information. UC11 is a part of requesting the model page, so figure 27 explains the process and then a download button is available in the model page. 39 CHAPTER 5. DESIGN AND ARCHITECTURE To achieve this set of functionalities described in the use cases, the sequence diagram in figure 31 showcases the steps needed and the order to be executed by the system. All three main pipeline components are present and represented inside a rectangle which is an entity. The sequence starts with the model training engine submitting a trained ML model to the Model benchmarking engine. This system indexes the incoming model in a database to be picked by the benchmark engine to be evaluated. The correct dataset tag is in the model dataset, so a request for the evaluation dataset corresponding to the tag is sent to the data ingestion engine, which returns the data, the benchmark is performed, and the resulting metrics are stored in the database. Finally, all the data in the database is available to consult in the end-user’s dashboard. Figure 31: Model benchmarking engine internal process With all the automated steps described in figure 31 a pipeline introduced model is benchmarked. However, one of the use cases implies that the user can also occasionally insert models that weren’t produced by the pipeline. The sequence is described in figure 32. It is important to understand that a model from the pipeline comes with predefined metadata, some necessary and some optional, that should also be present in manual inputted models, or else the system may be incapable of performing an automatic benchmark. 40 5.3. SOLUTION ARCHITECTURE Figure 32: Model benchmarking engine, manual inputted model An important use case is to quickly produce value from the data stored on the database to the enduser. The dashboard must provide the tools to parse the data and display it so the data scientist can see and perceive the information as efficiently as possible, for instance, finding the best performing algorithm for a specific use case or the best algorithm in a specific dataset. 5.3 Solution architecture Inside the model benchmarking engine, three main components are designed to answer the project requirements and use cases. This section specifies and explains the design choices associated with each. The first component requirement is to have a storage solution for the data associated with the system, the database component is designed specifically for this problem. The second component requirement is to have a benchmarking engine capable of managing all the operations related to a machine learning problem, fetching, scheduling and updating benchmarks for each model automatically without user input. The third component requirement is to have a full web dashboard capable of answering all the questions about the collected data simply and effectively. The dashboard container accomplishes this. To understand the general scope of each subcomponent in integration with the pipeline, the building block diagram in figure 33 explains the scope in three layers, each diving in more detail. In layer two, we have a more detailed look into the three components that together compose the model benchmarking engine. They are the database solution, benchmark engine and the dashboard. Finally, in layer three, we have a more detailed look into each component. The following subsections explain in detail the functioning of each subcomponent. 41 CHAPTER 5. DESIGN AND ARCHITECTURE Figure 33: Model benchmarking engine, building block diagram 42 5.3. SOLUTION ARCHITECTURE 5.3.1 Database solution, design, and architecture The first component is responsible for the system data storage solution. When considering a database solution, many factors are involved in the type and structure of data stored and the optimization of each database engine. The following question help to decide on which solution to choose: • Is the data scheme fixed or will it change? • Is there relations between different tables in the database? • How much data will the system store? • Does the database engine scale as required? • Is the database engine open source or does a licensing agreements with the company exists? Following these questions, the solution can start to be formulated. This problem is defined in an objectoriented language. The class diagram in figure 34 helps to understand the data structure and relations that exist. The scheme is expected to be mostly fixed so that no significant migrations will happen to the database with time. Most tables have relationships between each other, meaning a relational database engine will be ideal when only considering this fact. Full data size isn’t expected to be an issue, as actual model files will be stored independently. After consideration, the PostgreSQL database engine is chosen as it meets all the requirements. However, models files still need an appropriate storage solution. This solution is an independent object storing service called minio. Minio is an open-source, high performance and cloud-based storing solution that provides all the requirements for all pipeline components, so all the data lake brute files will also be stored in a minio bucket. Furthermore, since some datasets can reach multiple gigabytes of size, requests need to be processed as fast as possible, minio can write/read at up to 183 GB/s and 171 GB/s respectively. 43 CHAPTER 5. DESIGN AND ARCHITECTURE Figure 34: Benchmarking engine class diagram Following the diagram in figure 34. Each model has associated a unique author and an ’ explainability ’, which is where the different model hyperparameters are stored. Last but not least are the benchmarks performed on the model, with each benchmark having associated a dataset and its SHAP explainability charts. Log tables are also present to store system operations, malfunctions and exceptions. The data structure is not very complex but achieves all requirements, and with a unique model file, the component should be able to fill all this information automatically. The final consideration is for the interoperability standard between the database and the object storage solution, the model name works as the unique ID for each model and is stored in both storage systems. When a system wants to pick a given model, it asks the file storage solution for the model name stored in the relational database, and it returns the model file. Since model names must be unique the system guarantees the correct functionality. Finally, after a model is trained in the previous steps of the pipeline, it is inserted into the object storage solution. Then, an automatic process is scheduled to run in a given time interval which checks the models inside the object solution bucket and the ones indexed in the database. If new models are found, they are added to the system. The database solution is made of two distinct components as discussed above and shown in figure 33 on level 2 of the diagram: •Database engine - PostgreSQL 44 5.3. SOLUTION ARCHITECTURE The chart in figure 40 showcases the key metric to understand the raw output of the system in terms of benchmarks, the metrics benchmarks per minute is referenced in the chart as BPM, as can be observed, benchmarks per minute are at 0.51 in the single-threaded test, using two threads the value rises to 0.90 and then 1.19, 1.39, 1.56 and finally 1.67 using all six threads allocated to the problem. Further increase in threads is not sustainable on the hardware side, and returns are already diminishing, indicating that the parallelization process is optimized for the hardware available. Figure 40: Benchmarking engine scalability comparison, thread BPM comparison Using this scaling technique, the system now has a speed up of 327 percent comparing to the baseline single-threaded architecture, which significantly improves the response capabilities that it provides to continue to evaluate machine learning models, further in the future, if the system needs more scaling, a horizontal scaling approach might be required, which implies adding computational devices. 5.3.3 Web engine architecture The final subcomponent is the web engine, which is responsible for providing all the information collected by the system to the end-user in the simplest and most informative way possible. This goal is achieved by having a web framework handling the backend data and a powerful frontend dashboard to showcase all the information. All the data is fetched in the database solution. This means that no communications occur between the benchmark and web engines, which simplifies the architecture. The technologies to develop the solution are as follows: 51 CHAPTER 5. DESIGN AND ARCHITECTURE •Django: Django is a powerful python web frameworks that allows the creation of large scale web applications, it also has a powerful ORM database connector that works natively with the chosen SQL database solution and simplifies the querying process. •Bootstrap: Bootstrap is a powerful HTML, CSS and JS library that allows the creation of interactive and dynamic web pages, it is specially powerful to create dashboard like applications which are the goal of this application. The architectural pattern is the Model-View-Controller, which separates the application into three logical components, allowing for greater modularity, easier maintenance and a bigger upgrade path for the future. Figure 41 showcases the component structure and the connection to the storage solution. Figure 41: Web engine subcomponent explained 52 5.4. DEPLOYMENT OVERVIEW 5.4 Deployment overview The deployment of the application showcases the requirements and techniques used to deploy the application and the hardware utilized and required to guarantee the correct functionality. The figure 42 is a deployment diagram and displays all three main components in the pipeline. TCP-IP connects the model benchmarking engine with the data ingestion engine and the model training engine. Inside the model benchmarking engine, there are three main docker containers, each running a subcomponent. Communication between subcomponents only happens with the storage solution. The final users or the system administrator interacts with the system using a web browser and establishing an HTTPS communication that displays a frontend dashboard. Figure 42: Application deployment diagram 53 CHAPTER 5. DESIGN AND ARCHITECTURE 5.5 Summary Now that the architectural design for the application is built, there is a solid ground to start the development process, the architecture for the whole Atlas system is completed, the general overview can be seen in figure 43. Figure 43: Atlas final architecture 54 C h a p t e r 6 Results demonstration and discussion The focus of this chapter will be to demonstrate the web dashboard, all the information displayed in the dashboard is computed in the other components as shown in the previous section without additional user input. This is key for the system automation and time-saving benefits, as it allows the user to have its models continuously benchmarked without performing any action in the system. The initial page and the first interaction with the system is the default front page. It is designed to show key information about the model benchmarking engine and the Atlas pipeline and must be intuitive and straightforward, such as allowing the user to use the system without external help. Figure 44 demonstrates the initial page with multiple numbered rectangles highlighting different information available, Area 1represents the side menu that gives the user quick access to all major functions available. Area 2contains information about the system state like models loaded, benchmarks executed and pending an execution, number of different datasets present, and the number of the different authors who submitted models to the system. Area 3are quick access tables with the latest five performed models and the latest five performed benchmarks. It contains the name of the model, the author that created it, date of submission and a benchmark state with the latter being red if the benchmark is not yet performed or a green button if it is available, this allows the user to find its latest submitted model with as few clicks as possible. Last, area 4represents the Atlas pipeline status, since the pipeline depends on all components working properly, it is important to have all components’ status readily available and easy to obtain. In the initial page and following the side menu, the user can choose the models page, benchmarks, datasets, and the alchemy page which will be explained shortly. The model page is provided as shown in figure 45, most columns are self-explanatory and contain relevant model information, the last two columns provide information about the benchmark state and explainability state, if the icon is grey, the system hasn’t computed the information yet. A red icon means that some content may not be possible 55 CHAPTER 6. RESULTS DEMONSTRATION AND DISCUSSION Figure 44: Initial system webpage to obtain, and a blue icon indicates that all information is available. Clicking the button will take the user to the respective page. All information in the table can be ordered by column, searchable by text and pagination customizable by the user. Figure 45: Model listing webpage Similar to the model page, the benchmark page shown in figure 46 shows the benchmarks performed by the system, the main difference is that a model can have one or more benchmarks associated so this page will show each benchmark independently, the user can also access its dashboard and see all the information available. The dataset’s page is demonstrated in figure 47, and it contains information about all the datasets indexed by the system, the query tag, which is the human-readable identifier, and all the versions of the datasets present and the latest query ID correspondent to the tag. Finally, the last column takes the user 56 Figure 46: Benchmark listing webpage Figure 47: Dataset listing webpage 57 CHAPTER 6. RESULTS DEMONSTRATION AND DISCUSSION to the dataset page, shown in figure 48 and displays all available versions and a full statistical report about the given dataset. The dataset detail page is provided by the data ingestion engine, possible due to the tight integration between all the components in the pipeline and the achieved interoperability. it is of the highest relevance to provide dataset information, as users might want to consult the dataset used for training and evaluation for each model to replicate the results. The dataset information is very detailed with statistic information about a number of events, environment conditions and metadata information, giving the user a fully-fledged report on the data. Figure 48: Dataset detail webpage One of the main issues of having a fully automated system that computes all information in the backend without user input is the lack of knowledge about the status of those operations. To tackle this issue, a logging system was devised that catches checkpoints in common operations and stores their state; it can be seen in figure 49, the logging system also stores exceptions that happen in the system. To give the user the logging information, the admin dashboard was created. It provides all the latest logs from the system classified as info or warning, it is also important that new checkpoints for logs can be added with user feedback. This page also provides information about the system component status, a manual option to upload machine learning models that do not come from the pipeline, but the user still wants to introduce into the system, and a button to access the database management system which provides CRUD operations. Accessing the Manage database button will take the user to the page represented in figure 50. This page provides full control for the tables in the database, allowing the system manager to solve potential exceptions or faulty models that arrive in the system. It is expected that most hiccups can be treated by consulting the exception in the logs page and solving the underlying problem in the database management page. 58 Figure 49: Admin dashboard and log information webpage Figure 50: Admin dataset management webpage 59 CHAPTER 6. RESULTS DEMONSTRATION AND DISCUSSION The model benchmark results need an appropriate dashboard that is easy to interpret and delivers as much information as possible without overwhelming the user, it is also important to show the multiple benchmarks a model might have and appropriately identify them. Figure 51: Model benchmark page The model benchmark page is shown in figure 51. Each drawn rectangle in the picture references a different informational section. Area 1showcases model information like the dataset tag it uses, data window for prediction and even benchmark time, it also has a shortcut for the model explainability which will be explained later. Area 2is the metrics section, it displays all the metrics considered relevant by the data scientist for the specific machine learning model with its name and value. Area 3is the chart section that displays benchmarks results in time for each benchmark, this chart is crucial to understand the model evolution for each benchmark performed. Area 4is the section that displays all the other benchmarks performed on the model to give the user a good reference, the date of benchmark is the first column which allows to identify the timeframe for each benchmark. Figure 52 showcases the benchmark distribution section of page 51, it displays the labels predicted by the model and the actual ground truth value and is very useful to understand the model bottlenecks and if the training process needs to be improved. The model hyperparameters page is an important feature to have since it allows for other users to replicate and improve upon already good models, it makes each machine learning model reproducible, this page is displayed in figure 53, each box represents one hyperparameter, this information is dynamic and can work for any given compatible machine learning algorithm. Finally, the last feature is the model explainability, to achieve this functionality SHAP framework is used, which is leading state of the art in algorithm explainability. ML explainability is currently a hot topic and helps demystify the black box feeling about ml algorithms, it is specially important for two main purposes: 60