New Innovations in eIDAS-compliant Trust Services: Anomaly detection on log data

Master degree course in Innovation and Research in InformaticsData Science Master Degree Thesis New Innovations in eIDAS-compliant Trust Services: Anomaly detection on log data In collaboration with: Author: Marc Méndez Roca June 2020 Director: Dr. Francisco Jordán 2 3 Abstract TrustedX is a product that provides electronic signatures, it has been on the market for several years now, in result of that, market tendencies have changed and technologies too which made this product be on the same level as the competitors. This project tries to adapt an existing solution to the necessities companies have. To do so we proposed multiple changes, which have been divided into 2 parts, being the first one a part done by 3 UPC students (Arthur Bernal, Xiaolei Lin and Marc Méndez) with help of Dr. Francisco Jordán, the director of the project. The second one, being an individual part which will be only done by Marc Méndez. The common part is based on adding necessary technology, capabilities and functionalities that make this product able to work as a cloud multitenant system. This product will have the ability to sign official documents through the internet with the same validity as notarized signatures. The company had already the on-premise product developed, but now, instead of having an onpremise system we will create a new cloud-based multitenant system that has no location restrictions under the name of TrustedX as a Service (TXaaS). The whole TXaaS System is a common goal of this multi-part project. The specific part is the one that has strong relation to the specialization of the master, data science, in particular Deep Learning. Nowadays, a lot of products or companies use Machine Learning techniques to many different areas being marketing and business intelligence the two top areas of application. The proposal of this project is to develop a Deep Recurrent Neural Network-based Autoencoder for anomaly detection, which is able to learn large and difficult data patterns generated by the TXaaS. 4 Acknowledgments Being now at the end of this project I would like to look back and thank to every single person that has contributed to this project. I would like to express my sincere gratitude to my director, Francisco Jordán for the confidence given on me despite the few experience on the area, for the continuous motivation and encourage to keep improving day by day. I would like to thank Arthur Bernal and Xiaolei Lin for their cooperation to develop the project. As every group project, this could not have been possible with the help and dedication of each one of the members. They are workmates that you can rely on, doing a great job in every area they had to work, so more than deserved credits for them in this project. Thanks to Entrust Datacard Corporation for the opportunity provided to the students and the help of the workers of Barcelona, which have been helping a lot us to integrate and getting used to work on the company. Last but not least I would like to thank my family and friends for encouraging me to follow my road and being able to help whenever I needed them. Especially Marina which has been next to me all the time giving the motivation needed to keep doing what I like and always making me smile. 5 Table of contents LIST OF FIGURES ..................................................................................................................................... 7 PREFACE ................................................................................................................................................ 9 COMMON PART ................................................................................................................................... 10 INTRODUCTION ........................................................................................................................................... 10 Description ......................................................................................................................................... 10 Digital Signature and the eIDAS Regulation ................................................................................................... 11 TrustedX eIDAS Platform ............................................................................................................................... 11 Problem Formulation ......................................................................................................................... 13 TrustedX as a Service (TXaaS) ........................................................................................................................ 13 Trusted Digital Signing | Trusted Archive | Trusted Status Service ................................................................ 16 Trusted Status Service (AI/ML) ...................................................................................................................... 17 Goals .................................................................................................................................................. 19 General goals ................................................................................................................................................. 19 Specific goals ................................................................................................................................................. 19 Stakeholders ...................................................................................................................................... 20 Service and system consumers ...................................................................................................................... 20 Entrust Datacard Company ............................................................................................................................ 21 Students and the university ........................................................................................................................... 21 The market .................................................................................................................................................... 21 CONTEXTUALIZATION ................................................................................................................................... 22 Entrust Datacard ................................................................................................................................ 22 eIDAS Regulation ............................................................................................................................... 22 State of the Art .................................................................................................................................. 23 Competitors ................................................................................................................................................... 23 Innovation ..................................................................................................................................................... 24 SCOPE ....................................................................................................................................................... 24 Proposed Scope .................................................................................................................................. 25 The general Scope: ........................................................................................................................................ 25 Specific scopes: .............................................................................................................................................. 25 Challenges and Barriers ..................................................................................................................... 25 PROJECT TECHNOLOGIES AND INFRASTRUCTURE START-UP .................................................................................. 26 Basic Technologies and Concepts ...................................................................................................... 26 Cryptography, PKI and Digital Signature ........................................................................................................ 26 Web Services ................................................................................................................................................. 26 Web Applications ........................................................................................................................................... 27 Cloud Computing ........................................................................................................................................... 27 Distributed Ledgers and Blockchain ............................................................................................................... 27 Artificial Intelligence, Machine and Deep Learning ....................................................................................... 28 Proposed Technologies and Architecture........................................................................................... 29 Common Technologies and Infrastructure start-up ........................................................................... 30 Virtualization Infrastructure: VMware........................................................................................................... 30 Cloud Infrastructure: AWS ............................................................................................................................. 31 Database Infrastructure: Postgres ................................................................................................................. 32 Directory and Authentication Infrastructure: OpenDS and TrustedX ............................................................ 32 Event and Log Management Infrastructure: Splunk ...................................................................................... 33 Digital Signature Infrastructure: TrustedX ..................................................................................................... 34 DATA SCIENCE PART ............................................................................................................................ 36 REQUIRED KNOWLEDGE AND TECHNOLOGY ...................................................................................................... 36 Required knowledge .......................................................................................................................... 36 Machine Learning .......................................................................................................................................... 36 Classification and Regression......................................................................................................................... 36 6 Supervised and Unsupervised learning.......................................................................................................... 37 Anomaly detection ........................................................................................................................................ 37 Data Pre-processing ....................................................................................................................................... 37 Recurrent neural networks ............................................................................................................................ 38 Autoencoders ................................................................................................................................................ 38 Technology ......................................................................................................................................... 39 Jupyter Notebooks ........................................................................................................................................ 39 Python ........................................................................................................................................................... 40 Numpy, Pandas and Seaborn ......................................................................................................................... 40 Keras and tensorflow ..................................................................................................................................... 40 Amazon Web Services ................................................................................................................................... 42 STATE OF ART (DATA SCIENCE PART) ............................................................................................................... 43 Unsupervised Anomaly detection ...................................................................................................... 44 Anomaly detection for time series ..................................................................................................... 44 Why the decision to use Deep Neural Networks? .............................................................................. 45 Anomaly detection with Autoencoders .............................................................................................. 46 Anomaly detection in Time series using Autoencoders ...................................................................... 47 DATASET ................................................................................................................................................... 48 Data analysis and pre-processing ...................................................................................................... 48 Data scaling and transformation ....................................................................................................... 51 DEVELOPMENT ........................................................................................................................................... 52 Model definition ................................................................................................................................. 52 Model Implementation ...................................................................................................................... 53 Bias-Variance Trade-off ................................................................................................................................. 54 Cross-Validation ............................................................................................................................................. 55 Simple RNN vs GRU vs LSTM .......................................................................................................................... 56 Hyper-parameters ......................................................................................................................................... 59 Regularization ................................................................................................................................................ 65 Model Experimenting .................................................................................................................................... 66 System flow ........................................................................................................................................ 66 EXPERIMENT RESULTS .................................................................................................................................. 69 SimpleRNN ......................................................................................................................................... 69 GRU .................................................................................................................................................... 70 LSTM .................................................................................................................................................. 70 Error Distribution ............................................................................................................................... 71 Hyper-parameters optimization with Cross-Validation ..................................................................... 72 Problems encountered ....................................................................................................................... 74 Results analysis .................................................................................................................................. 75 Anomalies on the train set ................................................................................................................. 79 METHODOLOGY AND PLANNING .................................................................................................................... 80 CONCLUSIONS ............................................................................................................................................ 81 FUTURE WORK ........................................................................................................................................... 82 ANNEXES ............................................................................................................................................. 83 REFERENCES ......................................................................................................................................... 89 7 List of Figures Figure 1 - TrustedX eIDAS Platform Architecture ........................................................................ 12 Figure 2 - Current TrustedX Platform architecture ..................................................................... 13 Figure 3 - TXaaS general architecture ......................................................................................... 14 Figure 4 - TXaaS final architecture .............................................................................................. 16 Figure 5 - Comparison between AI, ML and DL ........................................................................... 29 Figure 6 - TXaaS general Architecture ......................................................................................... 30 Figure 7 - Structure for LDAP....................................................................................................... 33 Figure 8 - Splunk example of how data is presented .................................................................. 34 Figure 9 - TrustedX eIDAS platform basic architecture. .............................................................. 34 Figure 10 - Comparison between different types of representing categorical data ................... 38 Figure 11 – Autoencoder Structure ............................................................................................. 39 Figure 12 – Keras code to create a network ............................................................................... 41 Figure 13 – Equivalent code as figure 9 but using Tensorflow library ........................................ 41 Figure 14 - Relation of performance vs amount of data ............................................................. 45 Figure 15 - Example of Log .......................................................................................................... 49 Figure 16 – Structure of the 3D data ........................................................................................... 52 Figure 17 – Comparison between over-fitting, under-fitting and the perfect fit ....................... 54 Figure 18 – How affects to the loss function each type of Regularization.................................. 55 Figure 19 - Structure of K-fold Cross-Validation ......................................................................... 56 Figure 20 – Simple RNN cell configuration .................................................................................. 56 Figure 21 – LSTM cell configuration ............................................................................................ 57 Figure 22 – GRU cell configuration .............................................................................................. 58 Figure 23 – Most used Activation functions ................................................................................ 61 Figure 24 - MSE vs MAE ............................................................................................................... 62 Figure 25 - MSE and MAE equations ........................................................................................... 62 Figure 26 – Behaviour of different learning rates ....................................................................... 64 Figure 27 – Model with and without regularization ................................................................... 65 Figure 28 - Architecture of the System on AWS .......................................................................... 68 Figure 29 – Loss of SimpleRNN as nº of epochs increase............................................................ 69 Figure 30 - Loss of GRU as nº of epochs increase ....................................................................... 70 Figure 31 - Loss of LSTM as nº of epochs increase ...................................................................... 71 Figure 32 - Error distribution on a test set with non-regularized model .................................... 71 Figure 33 - Error distribution on a test set with regularized models (RNN left and LSTM right) 72 Figure 34 – Model loss without tuning hyper-parameters ......................................................... 72 Figure 35 – Model Loss with tuning hyper-parameters .............................................................. 73 Figure 36 – Model loss with tuned hyper-parameters and added regularizers .......................... 73 Figure 37 – Batch Normalization Layer ....................................................................................... 74 Figure 38 – Percentage of log distributed through day hours .................................................... 76 Figure 39 - Splunk Anomalies from 12/2019 to 06/2020 ............................................................ 77 Figure 40 - Time Distribution of anomalies ................................................................................. 78 Figure 41 - Time Distribution of anomalies with zoom ............................................................... 78 Figure 42 - Pie Chart of proportions of anomalies ...................................................................... 79 8 9 Preface TXaaS stands for TrustedX as a Service. TrustedX is an original product of the former Safelayer Company, now part of the Entrust Datacard Corporation (EDC). As a leading manufacturer of Public Key Infrastructure (PKI) technology, the company created TrustedX to consume digital certificates in electronic signatures. At this time, the innovation was to provide a secure remote system for applications to consume the service centrally through web services. This was revolutionary at the time as recognized by the European Commission and other European associations (see https://tinyurl.com/y7pk7joq and https://tinyurl.com/y6w8s598). More than ten years after TrustedX, we launched the TXaaS Project. It is an “Innovation” project under the Office of the CTO at Entrust Datacard and sponsored by the CIO (Anudeep Parhar). The context strongly encourages innovation: a small autonomous team of 4 people, 3 final master/degree students at the UPC (Marc Mendez, Arthur Bernal and Xiaolei Lin) and myself, professor at the UPC and Director in the Office of the CTO at EDC, as the link between the academic and business worlds. The common notable trait to mention about the team is that we are all passionate about technology. The whole team provides good informatics acknowledge and helped in different shared areas, as well as in new areas such as Artificial Intelligence and Machine/Deep Learning, Application Programming and Graphical User Interfaces, Cloud Computing, Distributed Ledger Technologies and Blockchain. What do all these technologies relate to TrustedX, PKI and Electronic Signatures? Only “Innovation” can answer this question comprehensively. Innovation, and Research and Development, is now a must in all businesses, not to be a leader, but to survive. The art of adapting to the rapidly changing technology landscape cannot be improvised. A product or a business can improve a little by updating an existing technology, however, they can revolutionize and explode by embracing a completely new technology. Where a company ranks between the two extremes defines it in terms of long-term market leadership. Innovation is a long-term continuous race that is cleverly imprinted in one of my favourites A. Einstein quotes: “Don’t expect different results by doing the same thing over and over again”. What is TXaaS? It can be seen as an Innovation project that seeks to improve TrustedX product, although what it certainly pursues is to revolutionize the way we can deliver value by adopting completely new technologies. F.Jordan Director Office of the CTO at EDC, PhD, Associate Professor at the UPC June 2020 16 Figure 4 - TXaaS final architecture In Figure 4 it is can be seen how the general architecture (or basics) that will support the digital signature service (Digital Signing) has been extended with new advanced services (Advanced part) to support the new services: - Trusted Archive that will allow the long-term storage of digitally signed and notarized documents using Distributed Ledger (DLT) technologies in general and Blockchain in particular. This trusted advanced service is also regulated in eIDAS, however, it presupposes the use of a technology other than DLT. However, there is evidence and we will also demonstrate in this work that the use of DLT / Blockchain is more efficient and reliable. - Trusted Service Status (Anomaly Detection) that aims to introduce analysis and prediction capabilities of the correct operation and use of the system in general, and of the digital signature service in particular. Using Artificial Intelligence (AI), Machine Learning and Deep Learning (M-DL) techniques, we want to infer information about the use of the service by consumers, in particular, we want to be able to define a “normal” behaviour of end users of the signature service through which we will be able to detect possible anomalies that could undermine trust in the system. For example, detecting possible impersonations of the signatories, thus providing a greater degree of confidence in the system and in the digital signatures generated. Having arrived at the final formulation of what we want to build in this project, the following section proposes a division of the problem into different common and particular parts that will be developed by the team involved. Trusted Digital Signing | Trusted Archive | Trusted Status Service The architecture and final proposal of the project has been divided into 3 large blocks related to the basic and advanced services described:  Trusted Digital Signing includes the new technological infrastructure made up of the TrustedX components (classic platform), and new LDAP and SIEM, as well as the new technologies introduced for Cloud Computing, API Management and general system administration through a single GUI. This block involves a common part of infrastructure and services for the rest of the blocks. 17  Trusted Archive introduces an advanced service that is supported through the additional use of new DLT / Blockchain and Cloud Storage technologies.  Trusted Status Service (Anomaly Detection) introduces an advanced service that is supported through the additional use of new Artificial Intelligence (AI), Machine Learning and Deep Learning (M-DL) technologies. Digging deeper into the project division, block 1) Trusted Digital Signing is much larger than the rest of the blocks. For this reason and for better organization, block 1) is divided into 2 parts: a. Common Infrastructure. A common part that includes the basic infrastructure elements (LDAP, SIEM, CA / RA and HSM) and the basic TrustedX platform (existing product). b. API / GUI Management. A part of new technologies and development focused on API management and the design and creation of a unique and uniform graphical user interface (GUI) developed exclusively on generated APIs. Finally, in addition to the study, design and development of different blocks, the proposal and development of proofs of concept, validators and demonstration applications are also included. The distribution of all blocks, both common and private, has been carried out with criteria of specialization and profile of the participating team, as well as academics for the defence of the final works. See “Scope” section for more details. Trusted Status Service (AI/ML) As explained before, we were not implementing a product from scratch but instead we were introducing a transformation of it into the market tendencies. The initial idea was quite general, search where we could apply machine learning techniques and what can we achieve from that. Then analyse what we can do in terms of self-benefits but also that is beneficial for the clients giving them a new features introducing innovative technologies with interesting functionalities. All of this had to be done with the data already available, gathered from the test application we have on the company, as, for the actual architecture of the software, we don’t have access to the user’s databases. So although the restrictions were quite big, we were not searching for a close term solution. We wanted to offer a long term solution that even if it’s not launched on short term due to the lack of real data, it could be a viable solution once we have recollected sufficient data. That’s the way we took, although we did not have access to implement our model with real clients’ data, we will have access in the future thanks to building a centralized SIEM controlled by us which will grant us, admin access to all logs. But to go through, we can take the data that was generated from the people using the test applications we have, which is pretty similar to what we will have in the future owing to the architecture of the new solution proposed. This applications are used on demonstrations to show the client what is the product capable of, even some of the logs of this apps are done by the clients when they use their test apps to sell to a 3rd party clients. These logs will be similar in shape, the only difference would be the number of logs per day received which should be much higher, and the processes which should be done with more logic than the test ones like repeating the same action to make emphasize in something. This can lead to some pattern recognition that should not be usual on real companies, but should not be usual. 18 To be honest, the solutions extracted would not be much different from the test ones, but is something that we will only see when training the model with the real data. Once TXaaS is launched, we will last some weeks until the model can be trained with the data. The idea is that, at the beginning as the operations performed on the test applications won’t be so different from the real ones, execute the predictions from the model that we already have, taking into account that the thresholds may need some readjustments. After having sufficient data to calculate a model from the data recollected from TXaaS, create a new model and compare the results between both to evaluate the difference. The miss of real data was not considered a problem itself as the project was proposed as a long term solution. What we had to figure out at this point, was what problem will data be used for, as there were a lot of possibilities after seeing the type of data and the features included in it. After brainstorming, we came up with some ideas. As the time went on and we were developing the common part, we were seeing more clear that, one idea was prevailing amongst the other ones. We initially thought about a fraud detection system to detect suspicious connections to the systems to avoid any intrusion, however, we made a deep dive into the area and came up with a more general idea that was the final proposal. Develop an anomaly detection system that was able to identify the logs that are quite different from the rest, by doing this, intrusions will also be inside this pack as they are considered as anomaly (they should not be happening often). This option has huge potential in long terms, as deep neural networks performance scale well with tons of data at long term. Moreover, the information gathered is double beneficial as we could ourselves see how users use our software in order to modify and make any upgrades on the software in a close future and recollect much more information thanks to having all logs through our centralized Splunk. Soon, this can be evolving into new machine learning algorithms that improve user experience. For example could be recommended applications based on the last applications used, or a system that estimates the month price based on the lasts months. Remember that when making the modification form on-premise to Software as a Service, we want clients to pay a subscription-based model, paying for the services they will use and adjust the final price according to it. Like AWS, pay for what you use, which can lead to a huge difference in price at the end of the month, so we think that providing clients tools to make easier the task is always useful. We considered anomaly detection as the number one priority due to the importance of it in a system that provides electronic signatures, being the difference into detecting or not some fraud. We had more options but this one was particularly special, not only for the size of the project as only the Machine Learning part is quite large, but also the challenge was very innovative using technologies and methods that most of them have less than 5 years of existence. 19 Goals General goals Regarding the general high-level objectives we can find:  Proposal and implementation of a service architecture and multi-tenant system.  Study and proposal of technologies that allow multi-tenant architecture as well as its integration for the construction of the system.  The final system must be scalable in all its aspects, both in the provision of services, volume of consumer users, integration of applications, support of an indefinite number of organizations, volume of transactions, etc.  The entire system must be able to be deployed in a private virtual infrastructure or in the public cloud (for example AWS).  Regardless of the technologies used, the system must include a unique and uniform API that allows consuming all the services, configuring the systems and subsystems, and accessing all the information that is generated.  The system will offer high capacities for information management, as well as analytics and business intelligence for its exploitation.  The system will be able to obtain the safety approvals and certifications that allow compliance with the current services legislation associated in general and with the eIDAS Regulation in particular.  The system proposal must minimize the cost of infrastructure and operation for the service provider.  The proposed system must allow for a subscription-based business model and must be prepared to incorporate a pay-per-use model in the future.  The system as a service (TXaaS) will offer or will be able to offer (because it is not limited by the new design) at least the same functionality that the existing on-prem system TrustedX already offers.  The resulting system will have all the security protections regarding applications and web services, including possible denial of service attacks.  Knowledge of new technologies will be studied, analysed and acquired from the point of view of the general innovation process of the company and not only from the point of view of its application in the specific systems and services proposed in this project. Specific goals Regarding to the high-level specific goals of the Trusted Status Service (Anomaly Detection) part, we proposed ourselves the following goals:  The data and use cases that may be derived from our system will be analysed and one will be chosen to implement in our system. (Not closing door of implementing the other options in the future)  Based on the data, different AI, ML algorithms will be analysed and the most appropriate one will be chosen for the construction of a model.  AI, ML technologies that are able to be implemented and work on our model will be studied and analysed and one will be chosen to integrate into the system.  An automated system will be implemented that allows inferences and predictions to be made about new data. 20  An automated system will be implemented to update the model so that the system learns from new data.  An API will be designed and implemented for the new service that will be integrated into the general API framework.  A visual application will be designed and implemented to validate and demonstrate the results of the new service. Stakeholders There are many stakeholders or interested parties in the results of this project and we have divided them into four groups: 1) Consumers of the service and system, 2) Internal, that is, the Entrust Datacard company, 3) The University and the students, and 4) The market. Service and system consumers End user who will have a very convenient, safe and reliable system to practice the electronic signature as opposed to current smart card-based systems (such as the DNIe) or USBs that force them to carry an additional device that can only be used in a personal computer (PC) and not always compatible with the host system. With the proposed system you can use it on any device, be it a PC, a tablet, a mobile phone, a SmartTV, etc. Application developers who introduce electronic signature and security based on signature identifiers in business, corporate or public use applications. These users will have a complete and uniform collection of APIs and support tools such as the developer portal. They will also have data analytics tools and business intelligence on all applications developed in operation. Organizations (companies or public organisms) consuming the service that will allow them to practice a subscription model, reducing costs for outsourcing. Administrator users / operators of the consumer organizations of the service, which will have a single graphical administration console and powerful tools for operation and management, data analytics and business intelligence. Administrators / operators of the TXaaS system will be able to perform their functions in a simpler and less expensive way, since it is a multi-tenant system and has graphical consoles and unified tools that give a unique and uniform view of the entire system through visualization, data analytics and business intelligence. Business-oriented users in the Marketing departments, Commercial, Business Development, Finance, Management, etc. They will have usable/ consumable data of the services generated by end users, developers, companies, organizations, etc. with the possibility of multiple segmentations such as temporal, geographic, certification authority, etc. Additionally, they may have a tool that automates the pay-per-use, that is, to account for the consumption of users and to make charges for consumption on a regular basis, for example against credit cards, etc. Compliance and security personnel (eIDAS Regulations, and others) both at the service provider level as they have a uniform system to which the security and compliance audit is standardized for one organization and is replicated in multiple organizations. For the staff of consumer organizations, most of the compliance and accountability process is outsourced. 21 Entrust Datacard Company The project allows you to improve an existing product (TrustedX) and create the bases of a new product (TXaaS) to be exploited. The project allows you to acquire new knowledge and mastery of new technologies that you can apply horizontally throughout the company, whether in other existing products or new products or services. The project allows you to improve your internal innovation processes, as well as use it as a tractor for new initiatives to continue the project or create new ones. The project allows you to expand and improve your image as a leader and innovator in the market due to the direct relationship with the University, as well as the dissemination of the project itself and the results achieved (TFG and TFM reports, demonstrators, etc.). Students and the university The University improves with business collaboration both at the institutional level and at the level of the teachers and students who participate in this type of project. There has never been so much competition at the university offer level, nor has there been so much demand from society in terms of specialization and especially at the ICT (Information and Communication Technologies) level, nor has there been so much speed in technological advances, which makes university-company collaboration a factor determining in both directions. The students and university professors who participate in this type of projects obtain from this collaboration a direct practical update of the business world that is essential to guide both groups in their initial career and in the focus of what society needs and demands. The university students who participate in this type of projects not only amply fulfil their academic commitment, but also obtain direct financing that helps them improve their quality of life, as well as promoting their professional project with many possibilities of joining the collaborating company. The market The market benefits directly from the results for the progress made in the project and above all for the creation of a superior competitive framework. The innovations and results obtained in this type of project can be disruptive in that they are a perfect framework for adventure and free experimentation without ties or preconceptions. The results of the project may affect the current competitive context, impacting competing companies. These projects are an incubator for new ideas and proposals that essentially seek differentiation (innovation) with the competition. 22 Contextualization Entrust Datacard Entrust Datacard 1 is a privately held company founded in 1969 with headquarters in Minneapolis, Minnesota, USA. Its origins are related to the financial sector and the creation and commercialization of hardware and software systems for the generation and personalization of credit, debit, etc. cards. Starting in the 90s, the company diversified and entered the government and business market with the generation of identity documents, travel documents, passports, as well as identity and control cards for companies. Until 2014 the company was called Datacard, and from this year the company changes its name to the current "Entrust Datacard" and expands its business strategy by adding the electronic and virtual world to the physical world of identity in which it operated until then. Datacard acquired in 2013 the company Entrust 2 , a world leader in PKI technology and certificates for the electronic identification of people, services and machines. In its digital business expansion strategy, in 2017 Entrust Datacard began its consolidation in Europe with the acquisition of several companies, one of them the Spanish Safelayer 3 at the end of 2018 Safelayer was a competitor to Entrust in the field of PKI and electronic identification, however, Safelayer had electronic signature technology, its own market niche in Europe and Latin America, and a group of experts specialized in PKI and eIDAS (European regulation) that complemented the Entrust Datacard group. This project began in September 2019 within the framework of Safelayer already within the Entrust Datacard group with the aim of innovating around PKI and electronic signature technologies by introducing new technological paradigms such as cloud computing, artificial intelligence and machine learning, Distributed Ledger and Blockchain, APIs and API management systems, responsive web applications and Rich Internet Applications (RIA) frameworks, etc. eIDAS Regulation Until the entry into force of the European Regulation eIDAS (Electronic Identification, Authentication and Trust Services) in 2014 the identification and electronic signature "safe and trustworthy" was associated with use of smart cards with cryptographic chip as in the case of Spain the DNIe (electronic DNI - www.dnielectronico.es). Since 1999, the European Directive on Electronic Signature 1999/93 EC 4 was in force, regulating and recognizing the electronic signature, legally equating it with the handwritten signature. The use of a physical smart card with a cryptographic chip and the technical complexity surrounding it and PKI technology manifested usability problems for users in applications, especially the Web. To this was added the boom of mobile devices and others such as WebTVs, 1 https://www.entrustdatacard.com/about/overview 2 https://en.wikipedia.org/wiki/Entrust 3 https://en.wikipedia.org/wiki/Safelayer_Secure_Communications 4 https://en.wikipedia.org/wiki/Electronic_Signatures_Directive 23 etc. in which initially there was no possibility of using the physical card and today it is still a minority and not without usability problems. The eIDAS Regulation is born as the answer to security and trust but, contrary to the previous Directive of e-Signature, taking into account the usability and accessibility by citizens. In this regard, Safelayer was innovative and an early precursor to this concept that was eventually regulated at the European level (see TrustedX European Innovation Award in 2007 5 ). This project wants to advance not only in the usability by the citizen of the security and trust of eIDAS systems, but also in the simplification of the integration and construction of eIDAScompliant applications by developers, as well as in the facilitation of the provision of eIDAScompliant services from both the consumer and provider point of view. Facilitating the deployment and way of offering applications and services using cloud computing, offering as-aService technologies and finding new paradigms for the improvement and construction of trusted eIDAS-compliant services is also a main objective of this project. State of the Art In the previous sections, the general context in which the project is located has been described. As it is a technological project, the context always has to do with technology, so in this section we focus on the untreated topics of 1) competition and 2) the state of innovation at the beginning of this project. Competitors A technological and business project (that is, a business project) cannot be proposed without indicating the reference competitors for which the development of the project will represent an increase or advantage compared to these. In this section we simply list very briefly those that we consider benchmark competitors. There are more, but we consider that they are perfectly covered with the selected group. Nor does this point intend to develop a detailed comparison of the competitors and conclusions of the competitive advantages to be obtained with this project. The simple enumeration of the competition already gives information on the current state of the art and where the project is located. A significant competitor is DocuSign 6 currently the largest provider of electronic signature services although most are under other non-eIDAS laws / regulations. This competitor is a benchmark in e-Signature as a Service and in the way that it facilitates the integration and construction of applications that use the e-signature. However, DocuSign is not European nor is it a benchmark in services under eIDAS, at which point our project can obtain advantages. In this same group we can incorporate Adobe with its AdobeSign service 7 . There is a group of competitors that we could classify as Trust Service Providers (TSPs) whose main offer is that of security and trust services based on PKI technology and digital certificates. 5 https://tinyurl.com/y7pk7joq 6 https://en.wikipedia.org/wiki/DocuSign 7 https://en.wikipedia.org/wiki/Adobe_Sign 24 Companies like DigiCert 8 , GlobalSign 9 and InfoCert 10 are our reference competence in this group. All of them have as their main service the generation and management of digital certificates, and all of them have joined the electronic signature service. However, the electronic signature service from a mentality or classic technological context, or in other words, modern technologies are not used and comparatively, very different from the strategy of competitors DocuSign and Adobe / AdobeSign, which we consider much more successful and therefore reference in e-signature. The world of technology and business is very competitive and must be viewed from a holistic and not a specific point of view. Competitors can be seen as such or as allies depending on the perspective adopted. In our case, due to our characteristics and the current competitive context, the companies (and their associated technologies) DocuSign and Adobe can be perfectly allied, however, the TSPs companies are direct competition. With the former, we find spaces where we can complement each other, but with the latter we do not. This project aims to innovate in the creation of a proposal that unseats TSP competitors by introducing the competitive factor found in DocuSign or Adobe. Additionally, proposals are also made that represent competitive differences with DocuSign and Adobe. Innovation Innovation for the generation of competitive advantage is the vehicle with which this project and any business project of any kind is approached. In the academic world, the objective of R&D and innovation also seeks competitive advantages, although the business and economic perspective is not always in mind, but the competitive nature is always and is also fundamental to the pursuit of objectives and achieving progress. Innovation, and Research and Development, is now a must in all businesses, not to be a leader, but to survive. The art of adapting to the rapidly changing technology landscape cannot be improvised. A product or a business can improve a little by updating an existing technology, however, they can revolutionize and explode by embracing a completely new technology. Where a company ranks between the two extremes defines it in terms of longterm market leadership. What is this Project? It can be seen as an Innovation project that seeks to improve an existing product, although what it certainly pursues is to revolutionize the way we can deliver value by adopting completely new technologies. Scope The challenges and barriers are quite common, although if any particular is identified, especially in new technologies and innovation, e.g. that the existing tools do not fulfil or do not function as expected, that the assumptions and theoretical proposals are not correct, etc. 8 https://en.wikipedia.org/wiki/DigiCert 9 https://en.wikipedia.org/wiki/GlobalSign 10 https://www.infocert.it/ 25 Proposed Scope The general Scope:  Propose, design and implement a complex “TXaaS System” made up of several parts that must be carried out jointly and separately (the 3 projects) that will result in several demonstrators, one general and others partial. Specific scopes:  Propose, design and evaluate the results of Trusted Status Service including the validity of this model to be applied in other areas like application-centred.  Analyse Trusted Status Service results and prove that by extension of the test data, the model can still work well in test environments. Challenges and Barriers  The technologies and tools selected at the end, do not meet the perspectives put on them. When working with very new technologies and tools, sometimes the choice is not appropriate and makes you delay or minimize the possible results or even not achieve the expected results.  Attempting to accomplish with the Regulation imposes certain technical requirements that condition the design and development of systems that may suffer a major setback if the regulation changes or becomes more nuanced, especially in young regulations such as that of eIDAS. In the case of eIDAS, there are still technical uncertainties that must be assumed and that could change.  Time. The duration of the TFG is limited and therefore, due to unexpected circumstances, the service may not be finished for the presentation of the project.  The risk of using new and innovative technologies could have influence if we get to a unexplored point or that we don’t know where to go. Is amazing to use new technologies and discover with them but we have to take into account that by being new we may find some adverse circumstances. There are other risks that are minor, but can influence the project:  Bureaucracy. In a large company, when requesting resources, it may take a long time to be accepted.  Data failure. Data loss in progress would be severe, but there are currently many resources that can be used to recover data in minutes.  Hardware failure. A hardware failure can cause delays in completing tasks, but working with multiple computers and storing copies in the cloud continuously mitigates this risk to almost zero, since, if one breaks, we can temporarily continue the other. 32 Database Infrastructure: Postgres Every TrustedX needs a database to store the information of users, logs, signature processes… As it has been commented before TrustedX has a small database included but since our objective is to make it multitenant we had to use a more powerful external database. We choose Postgres because is a powerful and tested technology as well as open source. We want to use a single database to store the information of all different tenants. Normally every TrustedX has a dedicated database but in this case all the TrustedX instances will share the same one. This made that designing a multitenant database was very important since we did not want to have concurrency problems when accessing the database and we wanted to assure the privacy of the information of every tenant. There are many different ways of achieving a multitenant database design but we had to find ourselves one that did not imply altering the behaviour of TrustedX. The solutions we considered where as follows:  We could create all the tables needed for our system only one time and then create a view for every single tenant that limited their view of those tables to only the rows that belonged to them. While this solution seemed fine and scalable, we realized we could not get this to work with the TrustedX. This is due to the fact that in order for it to work we should have a new row in every table that acted as an identifier of the user, so at the end we discarded this option.  The option we opted was to for every tenant creating a new schema inside our database and bind them to only that schema. On every schema we would then create all the tables that TrustedX needs. This solution is scalable, allows us to separate the information from every tenant and it was compatible with the structure that TrustedX follows nowadays. Directory and Authentication Infrastructure: OpenDS and TrustedX LDAP (OpenDJ) The Lightweight Directory Access Protocol (LDAP) is an open and industry-standard application protocol for accessing and maintaining distributed directory information services. The OpenDJ is an open-source Java Directory Server that supports the LDAP, and it is the derivative of OpenDS with a new LDAPv3 compliant directory service that can manage hundreds of millions of entries. As mentioned, TrustedX has its own directory for storing user information, but quite limited. For this reason, in this new TXaaS architecture, we incorporate OpenDJ for the centralized management of information for all users of the system and improve scalability, and we have configured for separating organizations safely (e.g. sample-domain and tenant-2) according to Figure 7. 33 Figure 7 - Structure for LDAP Authentication Delegated Process Servers TrustedX, as a product which is very flexible, and allows the incorporation of an external LDAP, even the application of embedded authentication adapts to the new situation. However, the objective of TXaaS (multi-tenant) is far from the functions of TrustedX (single-tenant). Therefore, we have created an application of authentication Delegated Process Servers (AuthNApp) to adjust to the new objectives multi-tenant of TXaaS that will be compatible with TrustedX SSO, in addition to including password management, such as SSPR (Self Service Password Reset). Event and Log Management Infrastructure: Splunk Splunk is a software to monitor, search and analyse data generated by another software. In our case it is used to work with the logs generated by TrustedX. The Platform already had a log management tool inside the web interface but was not as powerful as the Splunk. Splunk was designed to make easier to control the logs generated by the applications to detect patterns, provide measurement plots like amount of information sent a day in a simple way, and help the company to apply Business Intelligence. To be able to do that we had to send data in real-time to Splunk by deviating the logs to it via a TCP connection. With it is easier to filter data depending the provider, once having multiple TrustedX instances, Splunk will help us organize every data apart from being able to receive information from multiple sources. Also, it includes modifiable configurations to the users restricting accesses to certain data, giving the permissions desired and filtering between data engineers and audit. We have an admin which has access to everything inside, he also has the ability to add and delete data and create the companies. Each company will have an admin that will only have access to their data, and having the possibility to create reports, and graphical samples. This admin of the company is also able to create new users under him with the same or less privileges, for example the previously mentioned audit user that the intention of it is to check some data and reports. Giving to much information to this users is not needed and sometimes is also self-defeating as they will last longer to find what they need. As we can see in the Figure below, it provides a timeline to see the distribution of logs too see when the applications it has higher demand. In this screen we only have access to limited 34 number of logs as the user audit-1 is from tenant txaas. It is configured in a way that every certain period of time we make a report of the logs received so when asking for data the tool does not have to do it in real time as it is already done. NOTE. We have used a “Splunk Enterprise Trial license” and a “Free license” (see https://docs.splunk.com/Documentation/Splunk/8.0.3/Admin/TypesofSplunklicenses) which are sufficient to probe the concepts and test the system. A non-trial license will be required if the system is deployed in production or non-trial environments. [40] Digital Signature Infrastructure: TrustedX Following the successful scheme of the major Internet service providers (Google, Amazon, etc.) and social networks (Facebook, Twitter, etc.), the identity architectural model implements a user-centric paradigm in which i) users can explicitly give consent/authorize the transmission of Figure 8 - Splunk example of how data is presented Figure 9 - TrustedX eIDAS platform basic architecture. 35 their identity data to the applications they access or ii) the identity provider (IdP) defines administrative authorization policies that the user knows and implicitly gives consent for. The model adheres to the OAuth 2.0 authorization framework [1] with OpenID Connect 1.0 support [2] and to the SAML 2.0 Web Browser SSO Profile [2] for authentication. These protocols are based upon a pure Web infrastructure according to a RESTful paradigm and are today the technical base of the systems supporting hundreds of millions of users that perform billions of interactions and service invocations on the Web. They form, in fact, the framework behind the success of the Social, Mobile and Cloud IT triad. The architecture is a fusion of the classical 3-legged model already used in SAML, involving the user, the service provider (SP) and the identity provider (IdP), with OAuth's 3-legged model based around the user, the relying party (RP) and the authorization server (AS). The result is a combination of an IdP with an AS, with the SP and RP entities being one and the same with a different name and the user being the same in both cases. The advantage of the OAuth authorization framework is that it can integrate any authentication method, from the basic username and password to the high-security eID credentials and other authentication frameworks including SAML. The eSignature Provider (eSigP) component of the framework is basically a resource server (RS) specialized in PKI identity attributes, specifically, PKI keys and certificates used in electronic signature. The RS delegates the authorization process to the AS, which in turn delegates the authentication process to the IdP. Thus, obtaining access authorization for a PKI attribute is equivalent to doing so with a normal attribute. However, access to the "PKI private key" attribute does not entail returning the value of the attribute but rather the use of the "private key”, i.e., in a basic electronic signature process. For instance, in the case of RSA PKI keys for generating a PKCS #1 block [4]. The OAuth model allows applying in a standard manner a consent (authorization) process for obtaining access to the PKI key resource of the user for generating an eSignature, achieving in a natural way the explicit consent of the user who is signing. NOTE. Text extracted form [https://link.springer.com/chapter/10.1007/978-3-658-06708-3_6] Author: Francisco Jordan et al... See [24] for more details. 36 Data Science Part Required knowledge and Technology Required knowledge Machine Learning Machine learning is the study of computer algorithms through experience. Usually seen as a subset of Artificial Intelligence. The objective of it is by giving some sample data to an algorithm and “train” it with that data be able to memorize some patterns and replicate it in other datasets. Some of the most common uses are spam email filtering, computer vision, market prediction as well as much more complex applications. Machine learning is used in this cases and many more as it makes the job much easier than using conventional algorithms to solve those problems. Inside machine learning we have a lot of different algorithms, they differ in their approach, the type of data and the type of task they are solving. In regards of the type of data, two of the most used are supervised and unsupervised learning algorithms. [25] Classification and Regression Classification and regression are two of the major problems we face when dealing with Data science problems. Classification is the process of separating the data into multiple groups or categories. This type of problems have a finite number of classes where we want our data to be fitted in. Some examples of this type could be image detection, such as detection of some objects for example chairs. Either there is a chair on the image or not. E-mail classification into “spam” or “nonspam” and many more. The second type of problems are the regression ones. The output of this type of problems is not finite but instead it’s continuous. If the other problems objective was to classify into a class, this ones try to give a continuous value. Let’s be more precise, on the same example of spam, a regression problem would be instead of classifying it, give the percentage (0%-100%) of spam. In this case, when receiving a mail we obtain 71.8% probability to be spam. On the classification methods the normal way to measure the model’s output is by accuracy analysing the percentage of the correct classified data. On the other type, there are multiple ways to check how good our output is, for example two of the most used are Mean Absolute Error (MAE) or Mean Squared Error (MSE). But this it’s easy when you know the expected output (Supervised Learning) as it is calculated as the difference between the correct and the predicted. The real problem comes on how to evaluate if we don’t have the “correct” output. That’s one of the main problems when evaluating results of unsupervised learning. 37 Supervised and Unsupervised learning Mainly, machine learning can be divided into supervised or unsupervised learning. The first type is much easier in many aspects compared to the other one. For example, in our case, supervised learning would be data that already has a label that classifies it into anomaly or normal log. Although that would be ideal in our case, it is not like this. In order to have it, we would need that someone that understands perfectly the system, spends every day looking at every single log is reaching the SIEM and classify it. As this is not a viable option, we have to work with unsupervised learning. The problem with this type of learning is that we don’t have a criteria that says if we are giving a correct answer or not. Even if our model says it is an anomaly we will have to check manually to confirm because we don’t have any tag that says the class of this log. For this type of learning, we will need usually help of experts to confirm that our model works correctly and the results are correct. The main problems on large datasets start when false negatives exists as this will mean that to find them a check of all the dataset is needed. [26] Anomaly detection Anomaly detection also known as outlier detection is the identification of observations on the data that deviate from the normal behaviour. [27] Normally this rare events translate into something that shouldn’t be happening such as natural phenomena, frauds, potential market opportunities or critical incidents. As the technologies evolve, more and more applications and companies take decisions based on data. Reason why more and more companies invest money on anomaly detection systems for their applications which generate tons of data that are not used for anything apart of wasting memory. By applying this techniques would make the company aware of unexpected changes in the patterns, to be detected easier and faster. In large companies this could save them a lot of money. One clear example would be the Wall Street to be able to predict the movements of the market before the others, know when something strange is happening for example that the company is getting acquired by one of the largest companies in the world. This area has gained a lot of value in the last years and more and more companies are trying to automate anomaly detection system on their products. Data Pre-processing Data pre-processing is one of the most important steps on data mining. [28] Normally this process takes a major part of the project, so is very important to invest the time needed to get the best data possible as the results depend highly on having good pre-processing. Every dataset is different, each one of them will need different processing and also depending on the model we want to develop we will need to prepare the data according to it. Data cleaning is the process of removing all redundant information as well as detecting all the information which can be useful or not in our project, is very important and will be present in most of our machine learning projects. 38 Another important step would be normalization and transformation, being the process of preparing and setting the data for our model. Normalization comes to modify all numeric variables into a common scale while transformation is more modifying the aspect the data has. Normally in Neural networks One Hot encoding is used for categorical variables. Instead of having a column for each variable we set it for each category giving a value of 1 if that observation belongs to that category or 0 if not. Something like the “Figure 10”. This is done because sometimes the model can misunderstand the labels of the categories, the average between apple and broccoli is not chicken, and this will lead to an awful model. In order to avoid that, binary columns are used to avoid misunderstandings like giving more importance to a category just because it has a higher categorical number. Recurrent neural networks To understand the decision of why we used a recurrent neural network we first need to know what type of data we are using. Our dataset is composed of logs from a system, which are time dependent. Is not the same for the system to receive a login and after it a logout, than the other way round, which clearly will lead to an anomaly. So we needed a neural network that was taking into account the order of the logs. The recurrent neural network, also known as RNN are mostly used on translation, speech recognition and some other order-dependent cases. The particularity of this type of networks is that it remembers part of the previous information. Applied to our case will be that for example, the process of e-sign summarized is: Login, Sign, Logout. So our model remembers this is the typical order of receiving the information Autoencoders Autoencoders are a type of artificial neural networks mainly used for unsupervised data although its procedure is by applying supervised algorithms. [29] The way it works is a network that its input to its output are the same. It is divided into 2 parts, the first one is the encoder which transforms the input data into a code that represents the input with less features Figure 10 - Comparison between different types of representing categorical data 39 (compressed to contain most important information). The second part is the decoder which does exactly the opposite, from the code it transforms it to the output. Although autoencoders where already being used back in the 80’s, their applications back then was quite different from the ones used nowadays. When autoencoders were being started to use, the typical application was dimensionality reduction, similar to what PCA does. More recently, autoencoders are being started to be use on a lot more of application fields such as Information retrieval, Anomaly detection, Image processing (dealing with image noise) and others not that popular. The most common design of autoencoders is training to minimize the error such that we are able to compress the data to a certain size. Most of the autoencoders will define the Loss as a function such that it minimizes the Error between the input and the output similar to the equation below. ℒ(𝑋,𝑋′)= (𝑋 − 𝑋′)2 Where the objective is to minimize the difference between what is predicted and what is the input/output of the model. Another variation of the model would be the following one which is a design to not punish so much large errors in comparison to multiple small errors. ℒ(𝑋,𝑋′)= |𝑋 − 𝑋′| Technology Jupyter Notebooks Jupyter notebooks is an open-source web application that allows users to create and share documents that contain live code, equations and text. A lot of data scientists use it to develop their machine learning projects. The reason is very simple, this notebooks are very intuitive to use, the can show plots, text, script results, tables, etc. It supports over 40 programming languages as Python, R, Julia, Scala, etc. Also it’s not like the typical scripts of code, in this application, you can execute chunks of code repeatedly. Imagine you are testing a function and you wrote wrong a variable name, you don’t Figure 11 – Autoencoder Structure 40 have to do all the steps previously done but only repeat the function itself. This is very useful when you are working with big datasets where the load takes a lot of time or when we executed a model that lasts long time and after it we want to modify some chunk of code, we don’t need to execute again just only the chunk we want, so that’s why in jupyter you need to take care to not overwrite variables you may need after. Python One of the languages that can be run in Jupyter is Python, more exactly we will be using Python 3, as all the versions of it are open source code which help people interact with it. Python is one of the easiest programming languages to read and learn, but as well it is one of the largest languages thanks to the community contributions. [30] This programming language was chosen over others because the amount of libraries and frameworks related to machine learning it provides making every task easier. When using some of them, we speed up the time of develop a lot. There are frameworks to help data cleaning, pre-processing, model developing, data visualization, etc. In fact almost everything you will need to do on machine learning projects can be done with some of this libraries. It has one of the largest communities and a lot of people are posting their projects where you can learn from them and get tips on how to improve some procedures. The combination of python with jupyter notebooks takes the best part of both. This combination of technologies is used for lots of machine learning developers and recommended when someone wants to start into machine learning. Numpy, Pandas and Seaborn We will not go deep into them but a brief mention of what are they used for in the project. The first one is Numpy, which is mainly used for data computations as matrix operations. It is also included inside other libraries like keras or tensorflow. The second one is Pandas, is an extension to numpy for data manipulation and analysis. It provides a lots of data structures, reading and writing methods, join methods and many more. The last one is not as big as the others. Seaborn was designed for providing data visualization which is very important on machine learning. In order to detect anomalies and set the thresholds, this library makes the job easier and in comparison with similar libraries the amount of options and the simplicity of configuration makes it a viable option. Keras and tensorflow These are two of the most powerful deep learning libraries in python. In fact, we only use Keras in a direct way but Keras is run on a Tensorflow backend, so we considered that at least needed an introduction as they have strong relation. Tensorflow is an open source library for machine learning developed by Google mostly focused on neural networks. The way to use was very complex as the first idea of this library was to be used internally for private purposes, but once they made it public, there was a rush to make it as a more intuitive and easier way to develop models and that’s how Keras was created. 41 Keras was focused on solving the difficulties of Tensorflow and make it a lot easier. A high-level API helps a lot the development and people that are not experts on Tensorflow can also make use of this library. We will show a little example of a model and the comparison in code between Keras and Tensorflow. As it can be seen, apart from the amount of code that we get rid of by using Keras, it is also easier to read by having little knowledge of python and machine learning. To avoid long chunks or code that is difficult to interpret and check, we will be using Keras on a backend in Tensorflow as Figure 12. Figure 12 – Keras code to create a network Figure 13 – Equivalent code as figure 9 but using Tensorflow library 48 Dataset In this section we will talk about the dataset structure along with all the procedures that can be applied to data. Before starting we will have to remember that this project is under development which makes impossible to have the real data ready. However, a similar system (at least from the users’ point of view), has been running for the past years which was already been generating logs being pretty similar to the ones will be created with TXaaS. Our software, which allows users to do digital signatures, was designed to be integrated to any application. For example the normal use of it is a private web application were the company workers can login and upload the document they want to sign. The application returns a pdf digitally signed. Our modifications to the system would not make the applications from the clients stop to work, but instead keep working as if nothing has changed. This applied to the data means that we should not notice difference between the logs before actualizing the system and the logs after doing it. So, we will get the logs from the previous years and use them as the dataset for training our model. The only difference as already said is that we will be using test application logs as are the only ones we have access to, which is totally fine as the proof of concept will be more than able to verify if this method could be applied in real environment. First of all, data obtainment will be different from the rest of the obtainments once the product is actualized into TXaaS. The first one, as we had no access to the database where they were stored, we asked for them. Remember the lack of access to the data is solved when centralizing the SIEM, which will grant us admin access to any log generated by any tenant. At first, we noticed that, before understanding the logs, it was needed to understand what the system was doing as they were quite complex operations far away from what a normal application logs look like. After analysing with the help of Dr. Francisco Jordán (One of the creators of the software), we came out with a query that was able to remove redundant or unnecessary information that was helpful when analysing manually the logs, but was the same information in different formats. The reason to remove information was due to the excess of it as will be seen on the next section. In some project it has to be considered the possibility of having wrong data on the dataset. In our case all the data is generated by the same product and the logs don’t have measures or values that can be inputted in a bad way, most of them are just events or texts identifying what task was executed or the results obtained to do a task. As the TrustedX has passed recent regulations and several audits where they check that everything is working correctly, we can expect the correct behaviour of the logs. Moreover, they have been used for the company and all the clients over the last year and no problems have been reported over the lasts years. Data analysis and pre-processing An important step on the development of the model is the pre-processing the data, which in most of the projects it takes around 60% of the time to do, sometimes a bit more if the data did not have any type of preparation, in our case, was the second option and it took a lot of time to get the data prepared for applying machine learning techniques. 49 It is one of the largest steps and also one of the most important as the better we prepare the data the easier it will be for our model be able to detect the patterns. By doing that, not only we can make huge improves in terms of performance but also avoiding unwanted results. Scaling correctly the variables can avoid giving extra importance to a variable that wasn’t intended to have. It was seen when analysing the logs, that a lot of them were coming and not in an ideal way. In this system, Splunk is being used as the platform that will receive the logs and store them for the future extraction. For the training dataset a different procedure is done to obtain the data but, once we launch the product and in previous test versions, all data will be obtained through Splunk queries. The way we will be receiving it is in a JSON, thanks to Splunk, several tools are given to help visualizing, feature extraction and queries to modify the way data is presented. In other words, we could say Splunk helps on cleaning a little bit the dataset. During the process of pre-processing we found multiple problems that were not expected, some of them were only found on the data received from the database. During the testing of TXaaS, we didn’t find the same errors on the logs as solved it when storing logs by modifying that parameter. The problem we were talking about was that the log separator was the same character used on some of the descriptions of the logs, so when passing the logs into a csv, there was bad parsing. Some of the applications that are focused to work on JSON won’t find much problem as the parsing of JSON can be controlled in other ways like field delimiters ({}), but the Python library named “json” does not look at it but, looks logs delimiter like comma, semicolon or similar. To solve that, we could either develop ourselves a function to make what we wanted or modify the data according with function requirements which was much faster. To do so, we used regex to detect all the patterns that contained the character causing the problem and change it for whatever different character that was not generating conflict. As the reality differs a lot from the theory, another unexpected problem was found during the process, this one we already knew before finding it. The JSON had a variable shape meaning that, depending on the type of action the user was doing, different fields will be on the log making very difficult to find 2 similar logs. One of the ways we found out was to transform the JSONs into a large table where every row was a log and every column was a field, having some Figure 15 - Example of Log 50 columns empties as not every log had all the fields. But as we can see, JSON allows to have nested fields inside as it can be seen below in a log sample extracted from the Splunk. There is a function in JSON library from Python that from a JSON can be obtained a table where if there are nested fields like for example the id of the client, it stores in the table as client.id or client_id. Thanks to that function the process was much faster and we obtained a large table that could be uploaded and transformed in Splunk. In the first TXaaS version, all of these steps can be omitted as the data will be stored directly on Splunk and with no parsing errors. After all this steps we still not have the data prepared. The problem is that a lot of logs are repeated, for the same action we have the action itself and the call-backs, depending on the event we have, the log from each one of the components so it’s a lot of repeated information. We will delete all the information that is not important to keep just the necessary. This will help to improve the performance of the model as the initial data received was more than 350.000 different logs and a total of 365 columns. This was too much information to handle, but thanks to Splunk and the help of Francisco, we came up with a query to group all similar fields, delete duplicated information and avoid unnecessary information about the users’ activity. These logs were designed long ago and have never been modified, so some of them contained information that has no relation to the activity of the user but instead of the software and if everything is working as expected. As we want to target user anomalies we decided to avoid information that could confuse the model. After applying this query, we reduced the amount of fields into approximately 7.000 and 22 columns which is much simpler to deal with it. We spend some days checking manually and seemed that all the important information was contained here in a summarized way. This data after this huge reduce of the dataset is the last step that will be easy to read, as from this point on, we will be applying scales and transformations in order to improve the results on the Neural Network without caring about the readability of it. The aspect before any other transformation contains the following fields: 1. Request_ip 2. Service 3. Result 4. Signed 5. Cause 6. Domain_id 7. Client_id 8. User_id 9. Trace_id 10. Device_id 11. Sign_identity_id 12. Methods 13. Sso_result_level 14. Authn_details_target_acrs 15. Year 16. Month 17. Day 18. Hour 19. Minute 51 20. Second 21. Region 22. Country This dataset is the most pre-processed version that is still readable from the point of view of the user because, the modifications done here will make more difficult to read for a user. In the counterpart, those modifications will make the Neural Network able to converge faster and obtain better results. All this steps have been done to the train dataset but will be done in an automated way to the logs TXaaS will output. As those logs are sent directly to Splunk, all this transformations can be scheduled automatically to run whenever you want, making that when we access the data is already in the way we expect it to be. Data scaling and transformation After having the definitive version of the data prepared for the machine learning algorithm, the following step to be performed is to avoid having categorical variables in text, as neural networks aren’t able to process it. There are multiple ways of doing it but as mentioned on the Required Knowledge section, the best way to do it in a neural network is with OHE (One Hot Encoding). We will apply it to all of the categorical variables of the dataset. As python is the most used programming language for data scientists, there is already a function inside Pandas called “get_dummies” which does exactly the transformation from a categorical column into OHE. With the result of this function, we will obtain a lot of columns especially when having large number of categories but it isn’t a big problem as the neural networks don’t struggle with this. In fact, having the data in those format ensures that we are not giving more importance to any category just for having higher category number (usually when not doing OHE, we set categories by number, this makes higher numbers have more “importance” on the model when possibly does not have to be like this). And the last step which was one of the most complex steps was the process of how we will send the data to the model. As seen before, our model needs to take into account the temporal order of the logs. Recurrent neural networks peculiarity is that they work with time series. To follow this restriction we will need to transform our dataset into a 3D matrix (Logs x Fields x Temporal Aggrupation) and test different aggrupation in order to see which one fits better our dataset. The first and basic will be with user and day, although the ideal one would have been by user and action performed (for example: document signed). The main problem with this option was that some users tend to do multiple actions on a single iteration, others that did not logout as they keep the application opened and some more than open it and do nothing, making very difficult to delimiter the actions. To try to solve these problems we thought that if, by error we divide an action, this could lead to an unwanted anomaly which we could have avoided. The way to avoid it was by making the aggrupation by day and user. This aggrupation is the most similar to the previously mentioned that can be performed in an easy and robust way. The reason behind it is that, by doing this way, all the aggrupation will be under the same time window and although having more than one action inside an aggrupation, if an anomaly is detected it will influence the rest of the group by having higher error. Also this division was done after looking for logs timestamps and seeing that the majority of the logs occur during the 6AM- 52 10PM period, which makes our assumption that almost no one will have a process divided if we partition it by days. This was one of the most complex steps to perform as we needed to standardize the size of the matrix but the number of logs per day was not static so we needed to use a mask to make it work correctly. Imagine having a matrix like the one on the picture below, where “i”, the rows, are each one of the logs, “j”, the columns, are each one of the fields and “k” are each one of the dimensions, f.e. the first one will be “user1” “Day1” , “user1” “Day2” and so on. The problem before masking was that, following with the example on the Figure, if “k” has is 2x3 for a certain “k”, it will generate an error so we needed to add a row as a mask to keep having that dimension as 3x3 matrix. Development In this section it will be explained how was developed the Anomaly detection model using a RNN autoencoder. It will also contain information about the tools used and the final shape of the system including all components. Model definition To have this model working in an automated way we need all parts of the project to work as one, although for our part we will only use Splunk and AWS, this ones will not be working properly if one of the others fail. The flow will have the model itself and how we have been testing in a real environment will be explained in the section System Flow. In this section we will explain how the model was designed and the key points to have success on the execution of it. First of all, when doing machine learning projects in general we need to define what problems are going to solve with our experiments. The objective of the model is to obtain a group of logs, process it with the model and know if either it is anomaly or not. As every data science project, at the begging we will perform the same steps (making little modifications) until we achieve a model that is up to our necessities. With the training data we have, we will be training the algorithm and testing it until the results obtained are good for our purposes. If we hadn’t wasted the amount of time we did in preparing the data and choosing Figure 16 – Structure of the 3D data 53 the correct model to do this task, everything would start to fail and we would have no clue of why our model is not working as expected. Fortunately, we have spent tons of hours and the pre-processing we are pretty convinced to say it has been done in an accurate way. Our model will not be giving the final output, in fact, through thresholds we are going to decide either a value is anomaly or not. So will be our job to be adjusting the thresholds depending the results we are getting at each iteration of the model, at least until the model has sufficient data to be precise in the classification. The model (autoencoder) is going to make a dimensionality reduction and expansion, in this way we are going to obtain a dataset that is similar to the input. The reason behind it is that by doing this we will make the model focus on the important features inside the large number of them, so when making the data pass through a bottle-neck the information that best represents the log will be contained while irrelevant information will be omitted. If all of this process is done with a loss function that forces to minimize the error between the input and the output such as “Mean Squared Error” or “Mean Absolute Error”, the autoencoder will try to make the best copy possible of the input after reducing the data to the size of the bottle-neck. Once the predicted values are obtained through the neural network, we can calculate the error between them by one of the two methods mentioned above. This error will be a value that indicates how different the prediction is from the actual value. Once we obtained all the values, we could use this error as an anomaly value. The model was designed to be able to obtain the best results in the largest number of logs which have a very low percentage of having anomalies (<1%). As mentioned before, the logs extracted from the database was the use of the application during a year where we can assume that if there is any anomaly the number will be so low that during the process of generating the model, this values will not have sufficient weight to influence the model. Therefore, we can expect that the model will be designed to minimize the error of the logs that is more usual to find. With all of this said before, the model will have larger error on the logs that are anomaly as the weights of the features will make it obtain a 𝜒ℎ𝑎𝑡 that is not very similar to the input, making the error between them is higher. As the pattern are following this logs is not similar to the ones followed by regular ones making the algorithm not recognize the pattern resulting into a higher error. Model Implementation This section, next to all the pre-processing steps, is more than 90% of the project time. We will mention how we developed the project and what tools we ended up using because during the project, different tools and environments have been tested until we found the optimal one. As any project, when having low or null experience on a topic it is pretty normal to not obtain the best results at the first try. This happened to this project as there wasn’t a team of experts behind helping to develop it correct. There are multiple strategies to follow to achieve the best results. Apart from introducing the model we will be using, we will also explain the strategic decisions taken to obtain the best results. As the amount of decisions to be taken is quite long we will try to group them by topic, 54 as for example, there are a lot of decisions that are based on the parameter chosen (apart from the Hyper-parameters). Bias-Variance Trade-off When implementing a model, one of the aspects needed to take care about is, avoiding the under-fitting and over-fitting being both options equally bad. [32] On one hand, with under-fitting we understand as a model that it has not been trained enough and as a consequence the representation of the model is too simple to fulfil our necessities. It will under-perform and have higher train error and higher validation error. When this happens, we can say that we have high Bias. This type of error makes the model miss relations between features or relevant information that causes this high error. On the other hand, by over-fitting we mean exactly the opposite, we over train the model until the point that works very well with a particular dataset and fail when trying to fit with additional data. Everything here is trying to find the balance between both as none of them are good. We will notice we are over-fitting when, there is a significant difference between the errors of the train and validation/test datasets, being this last one much higher than the previous one. When this conditions are met, we find the case of high Variance, the problem here is that some data considered as noise (data that has a random behaviour and does not represents the whole dataset), are classified according making the model take a more complicated solution of what it should. To solve this, it does not exist a definitive solution other than test your model and try to find the balance between both cases. Every dataset is different and there are a lot of variables that affect it like the amount of features, the number of samples, the number of train steps and others. The figure below represents very well the relation of under-fitting, over-fitting and how the ideal model should be. Each one of these problems has a different approach on how to solve it. This is highly related to Bias-Variance trade-off. The objective in every project on these areas is to be able to obtain a model that not only does well with the datasets we have but also with the real data. In order to accomplish these requirements we need a model that has low bias and low variance, which corresponds to the plot of the correct fit. In the development of this project, we found on both cases, which in part is normal. It’s nearly impossible to find the balance at first try and when trying to solve under-fitting end up having over-fitting and vice versa. Figure 17 – Comparison between over-fitting, under-fitting and the perfect fit 55 When having under-fitting, there are some techniques than can be applied in order to reduce the Bias. First of all could be trying to increase the number of features if possible, the second one would be to remove noise from the data in order to have a more accurate model, this two options are not always possible to perform so if after trying them there is still under-fitting the last option would be to increase the number of epochs (“iterations” see Hyper-parameter section to get more information about epochs). If we are finding over-fitting in our model, we could apply some of the following procedures to help reduce the Variance a bit. As said before, not always all the options are possible to apply to get rid of the over-fitting, so we give more options just in case some of them are not possible to apply. As the first one, which is very situational we find increasing the size of the dataset. Another option could be the opposite as before, we could train less the model, or provide the model with early stopping methods that allow it to getting over-trained models. Another method which is very used is regularization, there are 2 types of regularization, depending on the model and structure of the data one is better than the other. There is a large list of strengths and weaknesses of each type but, to achieve the best model, a combination of both is needed depending the step. We can also try to avoid over-fitting by applying dropout. This means dropping randomly neurons of our network forcing the network to find other neurons do the job of the missing one to obtain the same result (well get a dive in on this topics in Regularization and Dropout in their respectively sections). Cross-Validation This method is used in almost every project of this kind to avoid that randomness split of data influences the model’s results. [33] Usually in most datasets there are outliers, mislabelled data or both. Cross-Validation is a technique used to assess the results of the train dataset to generate the model, are able to generalize to any other dataset. This method splits the whole data set into K different sets and prepares to make K iterations. At each iteration one different set is left out, and the other K-1 are used to train the model. This one left will be used as a validation sample. The general idea is that if we only perform one iteration we can’t ensure that the model generalizes well in all situations as we can be cheated by model selection. By making this iterations we make at least the model perform with K different datasets with K different Validation samples. Figure 18 – How affects to the loss function each type of Regularization 56 At the end what we want to obtain with it, is a model that will be able to perform better on unknown test samples. The results obtained by the average of the K iterations will be much more accurate to the real world results than what we will obtain by just making one iteration. The process performed to achieve these results is the one showed in Figure 19. Simple RNN vs GRU vs LSTM When implementing recurrent neural networks, we need to decide which type of it we are going to use. [34] Each type has a different way to calculate the outputs and obtain different results, it depends on the type of data we have, the length of the dataset, if we need more complexity. Sometimes it could be that we chose an option that has worse results only because we know that it will performs much better with model in terms of time and not that much of precision is needed, but that’s in very specific situations. To know a bit better how they calculate the outputs and in which situations they work well, we will introduce each type before testing all of them and choosing the best option for the type of data we have. Simple RNN As the “basic” recurrent neural network type we have Simple RNN. It is a fully-connected RNN that takes the previous output and multiplies it to the input of the actual, then data is passed Figure 20 – Simple RNN cell configuration Figure 19 - Structure of K-fold Cross-Validation 57 through a Tanh activation function (explained on Hyper-parameters section) in order to obtain the result. When using RNN, at least in the keras module, it is very important the usage “TimeDistributed” as a wrapper function. The inputs on this types of layers are 3D where one of the dimensions is the time step, so it is used like TimeDistributed(Dense(variable)), in order to apply the Dense(variable) layer to each one of the Time steps independently. SimpleRNN is the easiest way to keep information along time. The way this type of layer calculates the outputs is the following: ℎ𝑡= 𝐴𝑐𝑡𝑖𝑣𝑎𝑡𝑖𝑜𝑛(𝑊 𝑥𝑥𝑡+ 𝑊ℎℎ𝑡−1 + 𝑏ℎ ) 𝑦𝑡= 𝐴𝑐𝑡𝑖𝑣𝑎𝑡𝑖𝑜𝑛(𝑊 𝑦ℎ𝑡+ 𝑏𝑦) Where “t” is each one of the time steps, “x” is the input, “y” the output, “h” is a hidden variable (what remembers information about last observations) updated at each time step based on the inputs, “b” is a bias variable, some people omit this but both options are fine and “W” which is the weights assigned. In the cases where we have more than one hidden layer, the second layer is calculated from the “h” value from the previous layer and so on until we get to the output where the value is obtained from “y” which gets the information of the last “h” function. This last equation that gives us the output is the “TimeDistributed” layer. It is used in all RNN as a way to applying the same function to each one of the time steps. So, in fact, the formula of the SimpleRNN is only the first one, but they are usually used together to understand how to get the output once obtained the hidden variable “h”. LSTM The second type we are going to explain is Long Short Term Memory or LSTM. It is much more sophisticated than the previous type. One of the differences of this layer compared to the previous one, is the number of inputs, in this case we have 3 while having 2 on the previous one. Figure 21 – LSTM cell configuration 64  Learning rate: To explain in a simple way, we could say that learning rate is the size of the step that you take when moving to an optimal. It is a tuning parameter which setting it too small will take too long to converge while a step too large can pass by the minimum. As the figure shows, both options are bad, we need to find the point in the middle that allows us to find the best result in less time. Hyper-parameter optimization methods Sometimes is difficult to find the best hyper-parameters, especially when having large number of them with big range of options each one of them. That’s why where implemented some methods that would make the job of finding the optimal parameters a bit easier. We will start explaining the basic one know by Grid Search. This method was the traditional way of finding the optimal solution. It is an exhaustive search which could be done manually by implementing a loop, or with some library that already contained this function. The way it works is brute force, it check all the combinations possible and keeps the best one. Usually this method is supported by K-fold Cross-validation to generalize the result as much as possible avoiding dependence in the partition between train and validation. It’s very expensive in time and normally it’s parallelized as the different combinations are independent one from another. A more sophisticated solution is the Random Search, as the previous one, we define all the combinations but instead of trying them all, now we select them randomly. It can outperform Grid Search when having few number of combinations, although in terms of time is much better than the previous solution, the results may not be as good as before. This solution as before is also normally supported by K-fold Cross-validation for the same reason. The last hyper-parameter optimization that we will analyse (there are a few more methods for specific algorithms) is the Bayesian optimization. This method keep track of past evaluations that are used to build a probabilistic model mapping hyper-parameters to a probability of a score on the objective function. This method is based on the Bayes Theorem that directs the search in order to find the optimum. In a practical way, it builds an approximation to the objective function under the name of “surrogate”. This function can be calculated way more efficiently and gives us an estimation of the real objective function. The process executed is simple find the hyper-parameters that obtain the best results on the surrogate, apply them on the objective function, update with the new results and do again the steps until max iterations are reached. A big difference from other two examples is that this method instead of trying all combinations or selecting random, spends a little bit of time to select the proper step when choosing the next hyper-parameters before testing on the objective function. Figure 26 – Behaviour of different learning rates 65 Regularization Regularization is known as the process of adding information in order to prevent over-fitting. The need on emphasize so much on over-fitting is because it is very typical that non experts, are not taking those steps into consideration and when doing the experiments we start to notice strange behaviours and results that seem very good on the training dataset but when trying a test dataset they are awful. In fact by applying regularization techniques such as Lasso or Ridge, we are adding Gaussian noise to the objective function. Which may sound strange as we were trying to avoid noise as much as possible, but it’s not. By adding noise we make our model less precise which ends up having more error on the train set but making it more precise on the test one. As the Figure shows the blue model could be perfect for the train data, however by having this we will also have over-fitting. The idea of regularization focuses on making the model simpler so that it can be able to predict better test datasets. At the end, what we want to have is the green function, which ends up representing a model that is simplified and can fit more test datasets as it has been “generalized” for samples different than training. Ridge vs Lasso regularization Both consist in adding a penalty to the loss function, however, as we want to minimize the loss function, we will have to minimize the combination of the regularization with the loss function by finding a middle term. We are forcing the model to increase the error in order to fit better other datasets different from the train one. The difference between Ridge (L2) and Lasso (L1) is that, L1 shrink values making sometimes their value as 0, which will mean that this feature is not needed. This is why we say that L1 has feature selection while L2 does not because L2-norm produces non-sparse coefficients. Another characteristic is that L1 does not have analytical solution but L2 does, so allowing Ridge to be calculated more efficiently than the Lasso technique. Figure 27 – Model with and without regularization 66 Model Experimenting As the team that was developing the project had no experience on the field, the experiments setup that was run over the time has been changing a lot. As we were testing and checking which applications work well and which not we were adapting our way of working. The first tests were done in a Splunk package named “Deep Learning Toolkit” (DLTK). This software was chosen because at first our data is on Splunk and we did not have much idea of how the Splunk API worked to ask for data from outside. It worked with Docker containers that were running Jupyter Lab using a Python 3 kernel and we were still on the step of getting used to the technology. As the days were going on, there were some problems when using this software, the first one was the speed of execution. Splunk was run inside a virtual machine and DLTK was a container inside it, so the final speed of training was quite low. For the pre-processing part, worked well as the resources needed are not so high. The real problem appeared asking for all resources possible, at this moment, Docker crashed and we were not able to use it again, luckily we had saved several copies of the scripts before the accident. At that point we decided that, as we were using Python on Jupyter Notebooks we could migrate to any platform using the same, so we did that moving to a desktop environment where the speed of training was higher and he had all the files saved on the Desktop. The final objective of the project is to have it automatized on AWS, but they charge you when using their services so we found that for testing and developing purposes was easier to user our environment that was almost the same they have on the cloud. So we developed our model trained and tested on the Desktop and then once we have all configured upload it on the AWS to obtain make the automated system. Thanks to the large number of libraries in Python it can be easily seen the developments and the best configurations of the model. It provides lots of plots and graphics to detect and correct Errors such as over-fitting, adjust the thresholds or hyper-parameter tuning jobs. System flow The system we are developing is not only formed by the model but also for all the components that help us to achieve our complete system. It can be divided into a part that has to be done manually and the automated part. Both of them are at this point finished and the system is currently running without any human interaction apart from some supervision that the results are being generated and control when we receive anomalies. The manually part consists on the part related to the pre-process of the data received from the last year and the generation of the model. In this part is where we tune the model and make some adjustments on thresholds depending the results we are getting. We also prepare all the system to be automated, this process will only needed to be done once for almost all the steps except, the upload of the model that will have to be done every time we update it. The route to follow is the following, when receiving new data we will need to update the model including all the new data in order to keep it up to date. This updating process it is estimated to be done once a week or a month depending on the amount of data received, a minimum of data will be needed in order to make an update that has impact on the predictions. It is considered 67 that at the start there are fewer updates and as the number of clients increase we will be doing it more frequently. Once we have the desired model generated, we upload it to AWS Sagemaker and create and endpoint with this model. It could be done from the desktop notebook but AWS provides facilities of connections between the services making the process so easy to deploy an endpoint from inside itself that is not worth to loose time trying it from outside. This is the process that will have to be performed once every time the model is actualized, from this point on, it has been done once and it is automatically updated. In Splunk we have implemented some queries that are able to make the needed transformations and pre-processing that will have our data summarized, deleting all the information duplicated or that has no relations to the actions performed by the users. By having this we avoid some steps that we had previously done with the train dataset. Now once the TXaaS generates logs, they are instantly on Splunk already prepared to be consulted by the model. We have developed a Python script that is able to grab the data of the last period of time (could be either day or week depending the amount of data we will be having and could be changing along time according to the requests of the clients) from Splunk and make the last adjustments like one-hot-encoding or normalization. Once everything is done and the data is prepared to be sent to the endpoint model, we proceed to do it. The model through calculations obtains the predictions of it. We will proceed to collect the results and detect if there has been some anomaly on the last time frame. Anomalies are classified into 2 different classes apart from non-anomalies. The mild anomalies being labelled as yellow (reference to the traffic light) and extreme anomalies being labelled as red, which are the ones that with no doubt are far away from what are supposed to be. Finally, send the results to Splunk where will be stored. We store them there, because Splunk API is very intuitive to use and we had already a demo application that was using the logs from there to show some plots, so to do similar tables and graphics, will be almost no work other than changing the data you ask for. Automatizing all the processes, was already an easy step compared to the previous steps done. Although having scripts and the model endpoint configured, we needed to have all parts of the system running together and be able to avoid AWS charging you for extra usage apart from the essential. We had to optimize the use of the resources. The machines containing the scripts (EC2) are turned off waiting for the time to run the script. By doing this we optimize the cost of the system a lot, as the machine for doing the predictions process does not last more than 20 min, but just opening it we already pay 1 hour, so we save 23 hours every day (if we make daily predictions). When having multiple models for each tenant or application (see Future Work) then we will probably move to a more powerful machine able to parallelize all the tasks. We start the process by setting a timer on AWS CloudWatch which will schedule the “turn on” of the virtual machine that contains the scripts. This machine is stored in AWS EC2, is a version that was previously configured by AWS to have all python libraries related to machine learning installed so we save time by avoiding doing it ourselves. Once the machine turns on, we already configured it to automatically execute the script that is inside doing the jobs mentioned above. At the end when it has executed everything it turns off the machine to be the most economically efficient possible. 68 This schema shows the interactions of the whole system implemented in the cloud which has no user interaction on the execution, the only part which has any presence of users is when they want to consult their data/anomalies. Every certain time (for the moment will be executed once a week), the scheduler wakes up the EC2 instance with all the scripts of Deep Learning. It calls Splunk to get the data of the last period of time (Week), and accesses to Sagemaker Endpoint to evaluate them. After it, we convert results to standardized format (see Results Analysis). This formats will allow Splunk apply searches to detect anomalies and be able to easily detect when an algorithm or a trace had any error as we made identifiers to detect. Figure 28 - Architecture of the System on AWS 69 Experiment results This section will show all the tests performed during the development of the project. Some of those tests have been done in different platforms, computers and OS so we will not consider time a factor if not explicitly said it has been taken into account with similar conditions. All tests have been performed using Keras neural networks, under some constraints to have standardized results amongst all experiments. We used in all tests K-fold Cross-validations and the results seen will just be a mean of the K different results. We also used early stopping to avoid over-training the samples and this is the reason that some examples may have larger number of epochs than others. Some plots seem to be over training due to the small variation between epochs but the stopping method would only stop if we perform 5 epochs where we are not able to improve the best result. SimpleRNN Simple RNN are the first type of neural networks tested, as expected, this types of network does not have any gates that control the main problems encountered on RNN. This types of networks are difficult to train as they usually fall into Vanishing gradient problems. We found out during the testing on this layers that by not having control gates, sometimes we got errors when training because gradient got so small that made impossible to calculate the results. When we solved those problems we could see the results of this network and compared to the other types was a bit worse, although maybe it’s not noticeable. We can see that it has a really fast loss decay at the begging and seems that once reached a point it does not get much better. The lack of having variables to control gates sometimes made the gradient vanish to 0 so fast that after few number of epochs we were getting a loss of NaN. This was one of the examples obtained where we were able to train without falling into Vanishing Gradient. The mean loss was approximately around 0.15 which is big compared to the Figure 29 – Loss of SimpleRNN as nº of epochs increase 70 other results. The problem found out with these types of networks is that they struggle with longer sequences. In order to avoid this problem in the future we will ignore this type of network. GRU GRU and LSTM are pretty similar in terms of results, in theory GRU outperforms LSTM in terms of speed, but for the results itself seem that more or less are getting the same results in general which will force us to choose depending on the results of each model. By introducing gates it obtain better results and the reason why this type of RNN is faster is just because while LSTM use memory units, GRU just controls the flow of information without this unit making the whole process less resource-dependent. There are no demonstrations on which areas one outperforms the other the only proof is that majority of times GRU is more efficient. If we talk about the layer itself, are simpler to modify as having less parameters than LSTM. In our example we can see the curve generated by the model loss which lasted little more than 60 epochs to find the best model. LSTM LSTMs are very similar to GRUs in many aspects just that they do similar tasks but in a different way. The advantage that LSTM have over GRU is that they are able to remember longer sequences thanks to the memory unit they have. The have more gates to control the flow of information making them more sophisticated than the GRU. The dude that is generated here is to know if the sequences inside our dataset are short enough to be memorized by the GRU or if LSTM will make way better performance. Figure 30 - Loss of GRU as nº of epochs increase 71 We can see in the Figure below that the LSTM lasts longer to get close to the optimum. While GRU converged fast into an error approximated to the optimal one, LSTM does the opposite. The learning is slower but constant, and once we are close to the optimal, a bit more than 20 epochs are needed to find it. Error Distribution The results of the neural network were not directly the results we wanted to have. To set the correct thresholds at the begging we were using the plots that we will see below. This plots help to see the distribution of the error. The high values of the plot is where most of the data is focused, as expected almost every observation is around the same error except a few that deviate a bit. These plots were generated over the test dataset to check if the thresholds that we set previously by looking the train dataset were able to be applied on other datasets which may sometimes not correspond. We can even have no anomalies on some tests depending on the length and the variability of the data. Figure 31 - Loss of LSTM as nº of epochs increase Figure 32 - Error distribution on a test set with non-regularized model 72 - To perform this tests we manually generated a dataset that contained one extreme anomaly and the other samples were pretty normal in general. As it can be seen, there’s a huge difference on the same dataset when using regularized models or not. We prevent a bit of over-fitting the model making more generalized and able to adapt to different dataset. This test had one anomaly that was over the threshold of extreme, but sometimes could be mild depending on the model predicting it. While in Figure 32 we would have some mild errors and the extreme one (the one that was modified to be an anomaly). The regularized versions had all logs between 0 and 0.2 of error while having the anomaly on 0.56 which clearly could be classified as an anomaly thanks to the gap between them. Hyper-parameters optimization with Cross-Validation The Hyper-parameter tuning job has been done using Grid-Search combined with CrossValidation to obtain more robust results. We tested with LSTM as we got better results with it (Explained on the result analysis section). As the amount of parameters we were testing was a lot, the time it took to check every combination of parameters was approximately 3 weeks. This tuning job have changed a lot the results of the model itself. As it can be seen in the figure below this was the results we had before optimizing the parameters. We can easily detect that Figure 34 – Model loss without tuning hyper-parameters Figure 33 - Error distribution on a test set with regularized models (RNN left and LSTM right) 73 this model is not working properly, it has very low error in general including the first epochs where it is supposed to not be like this, thus has a huge decay of gradients and works with errors too low. This model is not working well, it can be seen that as long as we train more the error was not getting better but instead, its variation was a bit random. The second sample we obtained was a bit better than the previous one, the shape was starting to look like what it should, however, the error at the first epoch was too low and the curve was not smooth which could be an indicator that something was not going as it should. Was at this point that we realised that we needed to add regularizers and dropout to add the randomness to the model. Adjust the range of the learning rate of the Grid Search and try to obtain better results. The following plot will show how the hyper-parameter optimization step is one of the most important ones on a machine learning project. If done correctly, it can make a model that works very bad or has several problems like the one we were seeing at the begging of this section can be optimized until the point where it is able to make very accurate predictions. As it can be seen, the loss follows a descendent pattern, which makes the slope decrease as it gets closer to the optimum. Figure 35 – Model Loss with tuning hyper-parameters Figure 36 – Model loss with tuned hyper-parameters and added regularizers 80 different locations as they are selling the product and using the application to show the clients what are we capable of, but, at least this is not usual strange and so the model did well to assign it as anomaly. The user was signing a document at Prague (CZ) at 8 AM, two hours later was signing another document at Hesse (DE), which is more than 500km of distance but also at 12 AM, 2 hours later was making a login into one of the applications in Madrid (ES). Of course this can be done by taking a flight from Prague to Madrid via Hesse and the user could have used spare time to test a functionality, however we are not analysing if it is feasible or not. The only objective of the model was if it could be able to detect when something strange happens and in those cases worked very well. We still have to consider false negatives, which is a major problem on unsupervised learning. Probably it could exist some log that can be considered anomaly inside the dataset and has not been detected yet, but this algorithm is still testing and we are improving some features as well as adjusting the thresholds, however the amount of data will be increasing more and more once people start to use it so we expect that once we have more data we could be increasing the ratio of detecting more and more. Methodology and Planning The methodology followed on this project was agile. As the team member with more experience, my job was to organize and coordinate the team to work as 1. I did not have any impact on what they were developing, my job apart from the deep learning part and the common part that corresponded to me, was to make everyone on the team feel good and be working in an environment that they feel comfortable to do so. Also, next to the director of the project we were setting the internal dates when we wanted to have some part of the project finished. The way of working was quite special, even before the pandemic of COVID-19 started, we were already working from home. With that way of work, we found out to be more productive and flexible, being able to work at the time you wanted. For the meetings, once a week was scheduled to solve some problems we could have and, set the goals for the next week, although is true that some of those meeting where through Microsoft Teams (Similar to Skype). Mostly was dependant of the availability of the team. Setting checkpoints once a week helps a lot to improve by yourself having sufficient time to learn but also short enough period to do the jobs, which result in better individual performances. The jobs proposed were short and precise, always knowing what direction to advance. It’s better to do more steps but short than to do a large step but don’t know at where to go. Apart from that, we continuously interchange mails and even in some cases, short calls to solve doubts which will almost be like staying in the office. Feeling comfortable to work makes the job easier and also better results. This project as said before, is a combination of 3 individual parts each one independent from the others, and one common that is the join point of them. As the way to work, for the individual parts there’s not much to say more than individual learning and work, on the weekly meetings we shared what we have done in order to let someone suggest any improvement ideas or design wise. 81 The common part was a bit more dependent so we stayed more in contact and sometimes even meet in the office to check if what we were proposing was making sense all together. As this common part was done from Sep-Dec, we did not have any problems for organizing meetings. Fortunately, technology companies are ones of the less affected by the pandemic, we had already a way of work from home and the only thing we had to change was that instead of meeting in person we had to do it by the web, which is perfectly fine. Conclusions This project, proposed an innovative method to detect anomalies on the system that was also under development, the idea itself can be used in other systems to perform similar tasks. The main challenge of the individual part was to be able to predict anomalies on logs that have a certain time dependency. Having data with time dependency and, not only this but also variable size made that the solution we had to propose was able to solve all these problems. After researching we found out individual ways to be solving each one of the problems, however, we did not find almost anyone using all of them. The hard part was at the beginning, we did not even know what an autoencoder was, but as we were self-teaching ourselves on the topic we were finding possible solutions. Normally, projects before starting you have an idea of what you are going to solve and most of the time possible solutions of how to solve it. In our case this wasn’t happening this way, at first we had to figure out what we could do with the data that brings the project a bit of freshness as well as some content that could be sold to the clients as valid solution bringing innovative procedures. We figured out what we wanted to do before starting to make hypothesis, the basic solutions that came into the mind were not possible, all typical solutions like One-Class SVM, Classification models, Neural Networks where discarded by the nature of the data. After ending the project we could say that there are very few solutions that implement an anomaly detection system using a RNN autoencoder with variable size of the timesteps (each day have different number of logs). We know some of the technologies used are on the very early stages of it and need more testing around them, however, the results got on this period of time are pretty good and we have the purpose of keep improving them. We proved that this complex model, gave good results despite all the disadvantages. We also managed to implement a totally automated system capable of detecting anomalies without human intervention. About the project and being, both for the student and the company (Safelayer part of EDC), the first project of Deep Learning, the results obtained were pretty good and the projection it has is astonishing. We will have to wait until the product goes into the market and receives real data to analyse the performance there, but as far as we saw on the tests, the expectative is high. We will also add into Annex part some of the dashboards that can be generated with Splunk which provide a lot of information on the train and tests data that we considered interesting to have. However we will not be including the data as it may contain sensitive information about the users of the application. Apart from that, most of the information stored there is related to TrustedX actions which without the appropriate guides is difficult to follow. 82 Future Work There are some adjustments to be done in the future, starting from improving a little bit the model’s performance, this could only be made with more data and applying hyper-parameter tuning jobs to get better results. Also, to make the model more accurate we can delete from the train dataset the logs which behaviour is suspicious, if still any. This will make our model be more accurate on the predictions as the total model percentage of data that is anomaly is lower. Ideally we would like to have 0 anomalies on the training set as this would make the predictions very good but since the amount of data is very big is nearly impossible to analyse everything, when finding anomalies on the train set we just delete them on the next iteration of the model. It would not be the biggest upgrade for the model but better than nothing, by making little steps we can be improving overall results. This model was developed also as a proof of concept which results have been good so far. Knowing that, now we can use this structure to extrapolate and obtain other models that add more value to the product. The model we implemented has a global view and detects anomalies for the whole system. Some of the anomalies could be generated from applications that the usage is pretty low and the actions performed are quite normal but as the model is not used to see this actions can label them as anomaly. We proposed to develop models centred in tenants, users and applications. Obviously those models require that those entities have lots of data so these functionalities will only be provided to the ones that have certain minimum requirements. The general model works well for us, as we have the function of an admin to check the global system that everything goes well but, for a company this is not the best option as they want information about them not the others. These models would be centred on those entities making that the results are more accurate to them. It may happen that for the global model, some logs can be anomalies, but for that application-based model used by almost anyone the log is perfectly fine. A company is only using this application don’t want to receive an anomaly warning every time they do an action on this application. To do this we will need more time to collect necessary data to apply these detections at each one of the levels. 83 Annexes 84 85 86 87 88 89 References [1] D. Hardt, “The OAuth 2.0 Authorization Framework”, RFC 6749, IETF, October 2012 [2] N. Sakimura, J. Bradley, M. Jones, B. de Medeiros, C. Mortimore, “OpenID Connect Core 1.0”, February 2014 [3] S. Cantor et al., “Profiles for the OASIS Security Assertion Markup Language (SAML) V2.0”, OASIS SSTC, March 2005 [4] J. Jonsson, B. Kaliski, “Public-Key Cryptography Standards (PKCS) #1: RSA Cryptography. Specifications Version 2.1”, RFC 3447, IETF, February 2003 [5] Wang, Chi-Feng. “The Vanishing Gradient Problem.” Medium, Towards Data Science, 8 Jan. 2019, towardsdatascience.com/the-vanishing-gradient-problem-69bf08b15484. [6] Nicholson, Chris. “A Beginner's Guide to LSTMs and Recurrent Neural Networks.” Pathmind, 2019, pathmind.com/wiki/lstm. [7] Tiunov, P. (2019, March 7). Time series anomaly detection algorithms. Medium. Retrieved April 12, 2020, from https://blog.statsbot.co/time-series-anomalydetection-algorithms-1cef5519aef2. [8] AI, N. (2019, October 9). Build and deploy accurate deep learning models for intelligent image and video analytics. Medium. Retrieved April 15, 2020, from https://medium.com/dataseries/build-and-deploy-accurate-deep-learningmodels-for-intelligent-image-and-video-analytics-8ad755213c06 [9] Kaur Gill, Jagreet. “Log Analytics Tools and Automating with Deep Learning.” XenonStack, 26 Mar. 2020, www.xenonstack.com/blog/log-analytics-deep-machinelearning/. [10] En, Joseph Lee Wei. “Autoencoders: Neural Networks for Unsupervised Learning.” Medium, Intuitive Deep Learning, 18 Feb. 2019, medium.com/intuitive-deeplearning/autoencoders-neural-networks-for-unsupervised-learning83af5f092f0b. [11] V, Avinash Sharma. “Understanding Activation Functions in Neural Networks.” Medium, The Theory of Everything, 30 Mar. 2017, medium.com/the-theory-ofeverything/understanding-activation-functions-in-neural-networks9491262884e0. [12] Tian Yang, & Vikas Agrawal. (2016). Log File Anomaly Detection. Stanford University CS224d: Deep Learning for Natural Language Processing. https://cs224d.stanford.edu/reports/YangAgrawal.pdf [13] Swapneel Mehta, Prasanth Kothuri, & Daniel Lanza Garcia. (2018). Anomaly Detection for Network Connection Logs. arXiv.org e-Print archive. https://arxiv.org/ftp/arxiv/papers/1812/1812.01941.pdf [14] Maximilian Soelch & Justin Bayer. (2016). Variational Inference for On-line Anomaly Detection in High-Dimensional Time Series. arXiv.org e-Print archive. https://arxiv.org/pdf/1602.07109.pdf [15] Dominique T. Shipmon, Jason M. Gurevitch, Paolo M. Piselli, Steve Edwards. (2017). Time Series Anomaly Detection. arXiv.org e-Print archive. https://arxiv.org/ftp/arxiv/papers/1708/1708.03665.pdf