Towards a Dynamic Model for Credit Risk.

Towards a Dynamic Model for Credit Risk by Maria do Carmo da Rocha Sousa Ph.D. Thesis in Business and Management Studies, branch of Finance Supervised by: João Manuel Portela da Gama Elísio Fernando Moreira Brandão December 2015 i Biographical note Maria was born in Penafiel, Portugal, in 1980, the youngest of four daughters. She received her BSc in Mathematics, in 2003, and her MSc in Mathematical Engineering, in 2007, both awarded from the Faculty of Sciences of the University of Porto. She started her PhD course in 2010 at the Faculty of Economics of the University of Porto. In 2002, Maria started to work as a mathematics teacher. In 2004, she began her career in banking, as a risk analyst. By then, she joined the Change the Bank Projects in Millennium bcp, including the “Operational Centralization of OTP/NSF” 1 . Through 2009 and 2010 she participated in the application of the bank’s IRB (Internal Rating Based) approach, in Basel II. Until 2012, she also worked with the operations of the BCP group abroad in the role of models analyst, contributing to the IRB application in Poland. In 2013 she was promoted to technical coordinator of a Models and Analytics team in the Credit Risk Department, being responsible for the bank’s retail PD models management. Since the end of 2015 she is with the team of Corporate Financing. In 2006, she received an award of merit by the Banking Services Committee for the contribution of her team “Operational Centralization of OTP/NSF” to the business. In 2013, she won an international competition in Data Mining and Finance 2 , as part of the conference BRICS Countries Congress on Computational Intelligence. During her PhD she presented her work in the BRICS-CCI & CBIC 2013 Countries’ Congress on Computational Intelligence, in the 4th International Conference on Information and Finance - ICIF 2014, and in the EFMA Annual Meetings 2015. Part of her research is published in the Journal of Economics, Business and Management and in the Journal of Expert Systems with Applications. She acted as a referee for Evaluation and Program Planning and Neurocomputing Journals. Maria is married and is mother of two twin boys. 1 OTP: Outstanding Transaction Posting; NSF: Non-Sufficient Funds. 2 There were 60 teams registered and 4 finalists in the competition. The other finalists were the University of Technology and Economics of Budapest (Hungary), STATISTICA (Norway) and the Federal University of Rio de Janeiro (Brazil). The competition, open to academy and industry (financial, rating and analytics), focused on the application of credit scoring in the retail banking sector, and its goal was to promote innovative solutions to deal with the temporal degradation of credit rating models. ii iii Acknowledgements This work became possible due to the support of several different people and organizations. In particular, I would like to thank to Manuel Gonçalves and to Nelson Teixeira on behalf of the BCP Group, for their financial support. I am grateful to my supervisors, Professors João Gama and Elísio Brandão for their guidance over the last few years. I would like to acknowledge the anonymous referees, Professor Valerio Pesic and Professor Massimiliano Barbi for their constructive comments, discussion and reviews. Thanks to my family, the best in the world! Thanks to Bino and Tina, the most outstanding Grandpa and Nana. A special thank you goes to Jaime for his unlimited protection, patience and care, and to Pedro and Tomás, my little babies, for being the driving motivation in this long journey. iv v Abstract This thesis addresses the problem of credit risk assessment under changing conditions in financial environments. There are a number of serious challenges in this area that have not received sufficient attention so far. This includes adapting the rating systems to population drifts and stress-testing decision systems under extreme adverse conditions. Research evolves from a baseline methodology to new dynamic modelling frameworks, and is settled in two interconnected research domains: the first assumes that the rating systems’ frameworks should adapt to the changing conditions; the second deals with the influence of extreme circumstances in credit default and in the banking business. As part of our contributions, we propose new methodological frameworks for risk assessment, and we present renewed empirical measures for credit risk. Empirical studies use realworld financial databases, of which the most relevant is the Freddie Mac’s loan-level dataset, available online since 2013 in the follow-up to the subprime crisis in the United States (U.S.). In the first research domain we propose an adaptive modelling framework to investigate the two mechanisms of memory, short-term (STM) and long-term memory (LTM), in credit risk assessment. These components are fundamental to learning, but have been overlooked in credit risk modelling frameworks. We argue that different memory windows can be explored concurrently. This is important in the credit risk area, which often undergo shocks. During a shock, limited memory is important; at other times, a larger memory may be preferred. In the second domain, we have developed a stress-testing methodology under the international rules on capital requirements, the Basel Accords. We present the first study using Freddie Mac database, which describes and implements a risk-adjusted equity model to fund the loans lending. In this context, we analyze the impact of the probability of default (PD) and loss given default (LGD) in the return on lending, when subject to the most extreme adverse circumstances of the past. We contribute with a more realistic understanding of the behavior of the reference static models when exposed to major disturbances in the financial systems. vi vii Resumo Esta tese aborda o problema da alteração das condições nos ambientes financeiros para avaliação de risco de crédito. Há vários desafios importantes nesta área que ainda não receberam reflexão suficiente, tais como a adaptação dos sistemas de rating na presença de drifts e a compreensão do impacto de condições extremas nos sistemas de decisão. A investigação evolui a partir de uma primeira metodologia de base para novas estruturas de modelagem dinâmicas. O trabalho desenvolveu-se em dois domínios de investigação interligados: o primeiro assume que os quadros dos sistemas de rating devem adaptar-se às condições em cada momento; o segundo trata da influência das circunstâncias adversas extremas no incumprimento do crédito e no negócio bancário. Como parte das nossas contribuições, propomos novos quadros metodológicos na avaliação do risco e apresentamos medidas empíricas renovadas para o risco de crédito. Os estudos de caso utilizam bases de dados financeiros reais, das quais a mais relevante é a base de dados da Freddie Mac. Esta tem vindo a ser disponibilizada online desde 2013 no seguimento da crise do subprime nos Estados Unidos (EU). No primeiro domínio da investigação, propomos uma framework de modelagem adaptativa para compreender os dois mecanismos de memória, de curto e de longo prazo, na avaliação de risco de crédito. Estas componentes são fundamentais para a aprendizagem, mas têm sido negligenciadas nos quadros de modelização atuais. Defendemos que janelas de memória diferentes podem ser exploradas simultaneamente, o que é importante na área do risco de crédito, frequentemente sujeita a choques. Nas crises, a memória curta é importante; em fases estáveis, é desejável uma memória longa. No segundo domínio da investigação, desenvolvemos uma metodologia de testes de esforço ao abrigo das regras internacionais sobre os requisitos de capital. Apresentamos o primeiro estudo suportado na base de dados da Freddie Mac, que descreve e implementa um modelo de retorno dos empréstimos ajustado pelo risco dos ativos alocados a essas operações. Neste contexto, analisamos o impacto da probabilidade de incumprimento (PD) e da perda dado o incumprimento (LGD) no retorno dos empréstimos, quando sujeitos às circunstâncias adversas mais extremas do passado. Contribuímos assim para uma compreensão mais realista sobre o comportamento dos modelos estáticos de referência quando sujeitos a perturbações nos sistemas financeiros. xiv xv List of figures Fig. 2.1: ROC chart for two different classifiers. ........................................................................ 34 Fig. 2.2: Non-dominant ROCs. ................................................................................................... 35 Fig. 3.1. Actual meaning of scores and human misconception of the risk. ................................. 45 Fig. 3.2. Illustration of scores misalignment. .............................................................................. 46 Fig. 3.3. Freddie Mac database - gap between the interest rate and the FIX rate. ...................... 55 Fig. 3.4. Freddie Mac database - performance measures. ........................................................... 59 Fig. 4.1: Case study Brazil - internal default by month in 2009 and 2010.................................. 75 Fig. 4.2: Case study Brazil – model adjustment with the default central tendency. ................... 79 Fig. 5.1. Case study U.S. - adaptive learning windowing configurations. .................................. 97 Fig. 5.2. Case study U.S. - adaptive learning versus baseline static model. ............................. 104 Fig. 5.3. Case study U.S. - performance of the adaptive learning. ............................................ 106 Fig. 6.1: Illustrative IRB risk-weights for residential retail. ..................................................... 125 Fig. 6.2: Illustrative example of a cumulative default. .............................................................. 126 Fig. 6.3: Case study U.S. - Rates evolution in the period 1999-2012. ...................................... 131 Fig. 6.4: Case study U.S. - Gap subprime - prime rate, 1999-2012. ......................................... 132 Fig. 6.5: Case study U.S. - 30-year mortgage spreads in the period 1999-2012. ...................... 132 Fig. 6.6. Case study U.S. - Cumulative default rates used in each scenario. ............................ 135 Fig. 6.7. Case study U.S. - Expected returns, rw=50% and LGD=25%. .................................. 142 Fig. 6.8. Case study U.S. - Expected returns, rw=12.5×1.06×K and LGD=25%. .................... 143 Fig. 6.9. Case study U.S. - Expected returns, rw=50% and LGD=45%. .................................. 144 Fig. 6.10. Case study U.S. - Expected returns, rw=12.5×1.06×K and LGD=45%. .................. 145 xvii List of acronyms AIR Average interest rate AUC Area under the ROC curve BCBS Basel Committee on Banking Supervision BIS Bank for International Settlements BRICS Association of five major emerging economies: Brazil, Russia, India, China, and South Africa CRD Capital Requirements Directive EAD Exposure at Default EBA European Banking Authority EL Expected Loss EU European Union FED Central banking system of the United States (also known as the Federal Reserve System or Federal Reserve) FHFA Federal Housing Finance Agency FHLB Federal Home Loan Bank FHLMC Federal Home Loan Mortgage Corporation FIFO First in First Out FSA Federal Services Authority FOMC Federal Open Market Committee GAM Generalized Additive Models GLM Generalized Linear Models HAMP Home Affordable Modification Program HHI Herfindahl-Hirschman Index IRB Internal Rating Based IV Information Value LGD Loss Given Default LM Loss Matrix LTM Long-term memory LTV Loan to Value xviii M Maturity OECD Organisation for Economic Co-operation and Development PD Probability of Default PIT Point-in-time PSI Population Stability Index RBP Risk-Based Pricing ROC Receiver Operating Characteristic STM Short-term memory TTC Through-the-cycle UK United Kingdom US United States WOE Weight of Evidence 1 1. Introduction More than half a century has passed since credit scoring models were introduced to credit risk assessment and corporate bankruptcy prediction (Harold Bierman and Hausman, 1970, Altman, 1968, Smith, 1964, Myers and Forgy, 1963). Nowadays, in advanced economies, a high proportion of loans are automatically decided using frameworks where the credit score is the central, if not the unique, indicator of the borrowers’ credit risk. In the United States (US) the measure of risk FICO score is an industry standard, claimed to be used in 90% of lending decisions, to determine how much capital each individual can borrow and to set the interest rate for each loan. In the OECD countries, probability of default models, or PD models, are the cornerstone for the calculation of banks’ regulatory capital in the internal ratings-based (IRB) approach of the Basel revised Framework. For the retail exposures, the determination of the borrower’s PD, which in association with the loss given default (LGD) and the exposure at default (EAD), form the key input risk parameters for the minimum capital calculation. A credit scoring model is an intelligent system. The output is a prediction of a borrower falling into default in the future. Typical credit scoring models are developed from static datasets. Subject to context specifics, and provided that certain requirements of the methods are met, a timeframe for the development is delimited somewhere in the past. Then, by looking at historical examples within such timeframe, the model is designed using a supervised learning approach. The resulting model is then used, possibly for several years, without further adjustments. As a result, these models are quite insensitive to 2 changes in the financial environments, such as population drifts in periods of major financial distress. Our research has a three-fold objective. First, we attempt to introduce time changing economics and market conditions into credit scoring models. Second, we aim to outline a new and a more realistic modelling framework capable of self-adjusting and learning adaptively from the dynamics that distress or change consumers’ behaviour and their credit worthiness. The third objective is to evaluate the impact of extreme events on credit risk measures, and therefore in banks’ lending outcomes, under the current capital rules. We study and propose new dynamic modelling frameworks for credit risk assessment. Research is settled in two connected lines: the first assumes that rating systems should adapt to the evolving landscapes; the second deals with the influence of default rates’ fluctuations on banking business. In our research we start by documenting the underlying problem by focusing on the analysis of an extensive database that is the main driver of the ensuing study on dynamic models, the Freddie Mac’s single loan-level dataset, first published in March 2013. We investigate the dynamics and performance of over 16.7 million of fully amortized, 30-year fixed-rate mortgages in the US, granted between 1999 and the first quarter of 2013. In this investigation, we identify the frailties of the frameworks used in default prediction, to identify the implications to risk-based pricing and rating systems designs. Our analysis shows that, not only scores diminished their ability to predict defaults when the mortgage crisis had come to public’s attention, but also the real default rates by score are quite irregular over time. We therefore conclude that there is a link between scores, lending and default, mostly influenced by 3 the lending practices, although the mapping between these is not adequate. Existing credit scoring models are effective to sort individuals by risk, but they are not suitable to predict real default in each point in time (Sousa et al., 2015a). In light of the above, the core of our research focuses on investigating and developing an initial approach for dynamic credit scoring modelling, suitable to deal with the temporal degradation of static credit scoring models (Sousa et al., 2013). The model is firstly developed with a classical framework using a static supervised learning and a binary output. Then, using a linear regression between the macroeconomic data and the internal default in the portfolio, the output is adjusted by macroeconomic factors. This methodology jointly deals with the specific risk, which is captured from the data gathered by the time of the application, and the systemic risk, which is modelled with the regression. When applied to a portfolio of customers with credit cards, in a financial institution operating in Brazil, it produced motivating results over a one-year period. Subsequently, we extend this line of investigation by proposing a new dynamic modelling framework for credit risk assessment. The proposed model mimics the principle of films, in that it composes the credit scoring with a sequence of snapshots, rather than a single photograph. Within the new modelling scheme, predictions are made upon the sequence of temporal data, which is suitable for adapting to real default concept drifts, translated by changes in the population, in the economy or in the market. The new model enabled us to depict the two basic mechanisms of memory - shortterm (STM) and long-term memory (LTM) - in credit risk assessment. Central to this line of investigation is the idea that models can be improved when acting similarly to human learning and that STM and LTM are the driving components of that process. 4 The first line of our research argues that the credit risk assessment should evolve with the changes in the conditions underlying the models. However, when rating systems do not have such an ability, as those which prevail in the industry, banks’ lending policies and supervisory rules should be oriented by realistic evaluations of the financial business models. With this motivation, we developed a research pipeline, consisting of a stress-testing methodology, which is an important risk management tool used by banks and supervisors. In the second domain of our research, we present a first application of Freddie Mac’s database to stress-testing. Here, we propose a first implementation of a return on risk-adjusted equity model embedded in the contemporary capital regulatory framework, which can easily be replicated to other real-loans portfolios. Our research is pioneer in relating the FICO credit score, which from 1990 is the key driver of lending decision-making in the US, with the return on lending, through the last financial crisis. We found that much of the disturbances on the return on lending occur when the Loss Given Default (LGD) is sufficiently high. Below LGD=25%, empirical simulations show that lending to borrowers with lower scores produces positive returns in the long-run. Therefore, we claim that, for certain values of LGD, credit-cuts by score might have been an overreaction to the crisis. If sufficiently mature, these loans can boost portfolios compositions, because they are less exposed to early payments. 5 1.1. Contributions The main contribution of this research is a new modelling framework for credit risk assessment that extends beyond the prevailing credit scoring models built upon historical data static settings. The proposed modelling scheme is able to adapt more suitably to changes in the financial environments, like population drifts and shocks in the economy or in the markets. Our theoretical model is supported by empirical evidence. When the new modelling framework is applied to a set of 16.7 million mortgage loans granted in the U.S. from 1999 through 2013, it becomes apparent that new data consistently improves forecasting accuracy. We claim that existing models are improved when acting similarly to human learning, in which the STM and LTM are key mechanisms of cognition. While the previous line of investigation provides convincing results, there are some real business problems with adjusting models over time. First, lenders have little incentive to employ such models, because it is expensive and timeconsuming to build new scorecards. Models need to be internally tested and validated and then regulators need to approve them. Furthermore, regulators still promote models whose coefficients do not change over time. In this scenario, stress-testing methods are another important risk management tool used by banks and supervisors to measure the impact of some disorderly events on real systems (like financial crashes), although banks are still using unchanged models. We present a first application of Freddie Mac’s database to stress-testing, which can easily be replicated to other real-loans portfolios. Our proposal includes a first implementation of a 12 2.1. Credit scoring - from Fisher’s discriminant to machine learning The World War II promoted the first expert systems to evaluate a person’s credit worthiness. As credit analysts were called to fight, finance houses and mail-order firms requested them to write down their rules for deciding whom to give loans. Some of these were numerical scoring systems and others were sets of conditions that needed to be satisfied – expert systems. In the early 1950, Bill Fair and Earl Isaac created the first consultancy directed to finance houses, retailers and mail orders firms, making use of statistically derived models in lending decision. Still, until 1970 credit risk assessment relied most exclusively in human judgement. Connected with lending activities, this task was typically performed to support decision-making, following a specific credit application. The labour of the risk evaluator, often a branch manager, would involve the analysis of the likelihood of a customer repaying his debt, based on a number of clues that he could gather from a community leader or an external entity, such as the employer, a credit bureau or another lender. Main aspects that he would check would concern to the customer character and honesty and his ability to create wealth. A person with little or no history of credit, or without proven capacity to gather assets would have little or no chance of having credit. The depth of reasoning behind a decision could largely vary and final decision would much depend on the evaluator's mood and instinct. From customer application to the decision or credit granting, the process was usually slow (Thomas et al., 2002). The first approach to differentiate between groups took place in Fisher’s original work (1936) for general classification problems of varieties of plants. 13 The objective was to find the best separation between two groups, searching for the best combination of variables such that the groups were separated the most in the subspace. Durand (1941) brought this methodology to finance for distinguishing between good and bad consumer loans. The first method used in the development of credit scoring systems was discriminant analysis, mainly focused on credit granting in two categories of loans: consumer loans, and commercial loans. For an early review and critique on the use of discriminant analysis in credit scoring, see Eisenbeis (1978). The boom of credit cards demanded the automation of the credit decision task and the use of better credit scoring systems, which were doable due to the growth of computing power. The value of credit scoring became noticed and it was recognized as a much better predictor than any other judgmental scheme. Logistic regression (Steenackers and Goovaerts, 1989) and linear programming (see Chen et al. (2013) for a review) were introduced in credit scoring, and they turned out to be the most used in financial industry (Anderson, 2007, Crook et al., 2007). The use of artificial intelligence techniques imported from statistical learning theory, such as classification trees (Breiman et al., 1984, Quinlan, 1986) and neural networks (Malhotra and Malhotra, 2002, West, 2000, Desai et al., 1996, Jensen, 1992) have arisen in credit scoring systems. Support Vector Machine (SVM) is another method based in optimization and statistical learning that received increased attention over the last decade in research in finance, either to build credit scoring systems for consumer finance or to predict bankruptcy (Min and Lee, 2005, Li et al., 2006). Genetic algorithms (Chen and Huang, 2003), colony optimization (Martens et al., 2007), and regression and multivariate adaptive regression splines (Lee and Chen, 2005) have also been tried. 14 The choice of a learning algorithm is a difficult problem and it is often based on which happen to be available, or best known, to the user. The number of learning algorithms is vast. Many frameworks, adaptations to real-life problems, intertwining of base algorithms were, and continue to be, proposed in the literature, ranging from statistical approaches to state-of-the-art machine learning algorithms, parametric to non-parametric procedures. Lessmann et al. (2015) compare several novel classification algorithms to the state-of-the-art in credit scoring and provide an assessment of recent scoring methods that sets a new baseline to which future approaches can be compared. Practice actually suggests that rational behind choice is likely to be more shifted to palatability rather than to accuracy, translated by any performance measure (such as error rate, Receiver Operating Characteristic curve, Gini index, or other). As an alternative to using a single method, a trend that is still evolving relates to the use of hybrid systems (Lee et al., 2002), and ensemble of classifiers with which the outputs are achieved by a predefined sequence or rule, or a voting scheme. New concepts for adapting to changes (Adams et al., 2010, Pavlidis et al., 2012, Yang, 2007, Jung et al., 2015) and modelling the dynamics (Crook and Bellotti, 2010) in populations start being exploited in credit risk assessment. 15 2.2. Base concepts and techniques 2.2.1. Score formulation A credit scoring model is a simplification of the reality. The output is a prediction of a given entity, actual or potential borrower, entering in default in a given future period. Having decided on the default concept, conventionally a borrower being in arrears for more than 90 days in the following 12 months, those cases matching the criterion are considered bad and the others are good. Other approaches may consider a third status, the indeterminate, between the good and the bad classes, e.g. 15 to 90 days overdue, for which it may be unclear whether the borrower should be assigned to one class or to the other. This status is usually removed from the modelling sample; still the model can be used to score them. Applied to credit risk assessment, we are essentially considering a supervised learning problem with the aim of predicting the default y ∈ {good, bad}, given a set of input characteristics 𝐱∈𝐗. The term example, or record, is used to refer to one pair of (x, y). Supervised learning classification methods try to determine a function that best separates the individuals in each of the classes, good and bad, in the space of the problem. A robust model enables an appropriate differentiation between the good and the bad classes. It is achieved by capturing an adequate set of information for predicting the probability of the default concept (i.e. belonging to the bad class), based on previous known default occurrences. The model building is carried on a set of examples – training set – collected from the past history of credit, for which both x and y are known. The best separation function can be achieved with a classification method. These 16 methods include, among others, well-known classification algorithms such as decision trees (DT), SVMs, artificial neural networks (ANN), and Generalized Additive Models (GAM). Hands-on software packages are available to the user for example in R, SAS, Matlab, and Model Builder for Predictive Analytics. The accuracy of such functions is typically assessed in separate sets of known examples – test, validation or out-of-sample datasets. The idea is to anticipate the accuracy of that function in future predictions of new examples where x is known, but y is not. The output of these models is a function of the input characteristics x, which is most commonly referred as score, 𝑠(𝐱). This function has a monotonic decreasing relationship with the probability of entering in default (i.e. reaching the bad status). The notation of such probability is 𝑝(𝐵|𝑠(𝐱))=𝑝(𝐵|𝑠(𝐱))=𝑝(𝐵|𝑠(𝐱),𝐱)=𝑝(𝐵|𝐱), ∀𝐱∈𝐗, (1) where B represents the bad class, or in other words 𝑝(𝐵)=𝑝(𝑦=bad). Likewise, G represents the good class and 𝑝(𝐺)=𝑝(𝑦=good). Since 𝑝(𝐺|𝑥)+𝑝(𝐵|𝑥)=1, the probability of the complementary class comes as 𝑝(𝐺|𝑠(𝐱))=𝑃(𝐺|𝐱)=1−𝑝(𝐵|𝐱), ∀𝐱∈𝐗. (2) Among researchers and practitioners, a usual form of the score is the log odds score 𝑠(𝐱)=ln𝑝(𝐺|𝐱) 𝑝(𝐵|𝐱) and 𝑝(𝐵|𝐱)+𝑝(𝐺|𝐱)=𝟏 (3) 17 And so, the score may vary from -∞, when 𝑝(𝐺|𝐱)=0, to +∞, when 𝑝(𝐺|𝐱)=1, i.e 𝑠(𝐱)∈ . In this case, the probability of the default event can be written in terms of the score as 𝑝(𝐵|𝐱)=1 1+𝑒𝒔(𝐱) ,∀𝐱∈𝐗. (4) The most conventional way to produce log odds score is based in the logistic regression. However, other classification algorithms can also be used, adjusting the output to the scale of that function. Although a grounded mathematical treatment may be tempting to tackle this problem, it goes beyond the scope of this thesis. The basics of credit scoring and the most common approaches to build a credit scoring model are further detailed in the operational research literature (Thomas et al., 2002, Anderson, 2007). Recent advances in the area deliver methods to build risk-based pricing models and methodologies towards the optimization of the profitability to the lenders (Einav et al., 2013). In the following sections, we will describe some of the most widely used techniques in credit scoring development, being the Generalized Additive Models (GAM) the core method of our research. 2.2.2. Generalized linear models Linear regression models represent the linear relationship between a continuous response variable and one or more predictor variables (either continuous or categorical) in the form 𝐲=𝐱𝐰+𝛆, where 𝐲 is the vector of observations of the response variable, 𝑋 is the matrix determined by the 18 predictors, 𝒘 is the vector of parameters, 𝜺 is a vector of random disturbances, independent of each other and usually having a normal distribution. These models are appropriated for linear relationships between the response and one or more predictors. The 𝑋 matrix (and the 𝐰 vector) is usually extended with an additional variable, with all data points taking the same constant value on this variable. This allows obtaining linear relationships without the restriction of passing through the origin. The hypothesis of normally distributed regression errors is very restrictive. Generalized linear models (GLM), formulated by Nelder and Wedderburn (1972) are an extension of linear regression models, expanding the use of regression analysis beyond disturbances with normal distribution, which can be used when in the presence of nonlinear relationships. To understand GLM, first notice that the linear models have the following three characteristics: - The response has a normal distribution with mean . - A coefficient vector 𝐰 defines a linear combination 𝐱T𝐰 of the predictors 𝐱. - The model equates the two as 𝜇=𝐱𝐰. In GLM, these characteristics are generalized as follows: - The response has a distribution that can be normal, binomial, Poisson, gamma, or inverse Gaussian. - A coefficient vector 𝐰 defines a linear combination 𝐱𝐰 of the predictors 𝐱. - A link function 𝑓() defines the link between the two as 𝑓(𝜇)=𝐱. The most important case in which linearity is not enough is when y and  are bounded. The linear model is inadequate in these cases because complicated 19 and unnatural constraints on w would be required to make sure that  stays within range. Typically, the link function is used to transform the  into a scale on which it is unconstrained. The identity link specifies that the expected mean of the response variable is identical to the linear predictor, rather than to a non-linear function of the linear predictor. Although other link functions are possible (e.g. probit and the complementary log-log function), some common canonical link functions for a variety of probability distribution are given below in Table 2.1. Table 2.1: Typical settings in GLMs. Probability distribution Canonical link Function Meaning 𝒇() Parameter restriction Normal Identity 𝑓(𝜇)=𝜇 𝜇 real Binomial Logit f(𝜇)=𝑙𝑜𝑔(𝜇 1−𝜇) 𝜇∈(0,1) Poisson Log 𝑓(𝜇)=𝑙𝑜𝑔 (𝜇) 𝜇>0 Gamma Reciprocal f(𝜇)=1/𝜇 𝜇>0 2.2.3. Logistic Regression Logistic regression is an instance of the generalized linear models. It is similar to a linear regression model but it is suited to models where the dependent variable is dichotomous, that is, the dependent variable can take the value 1 with probability q or the value 0 with probability 1-q. This type of variable is called a Bernoulli (or binary) variable. 20 The independent or predictor variables in logistic regression can take any form. That is, logistic regression makes no assumption about the distribution of the independent variables. They do not have to be normally distributed, linearly related or of equal variance within each group. The relationship between the predictor and the response variables is not linear; instead, the logit link function is used: 𝑙𝑜𝑔 𝜇 1−𝜇 =𝐰T𝐱. Or, stated equivalently, 𝜇= 𝑒𝐰T𝐱 1+𝑒𝐰T𝐱. Fitting a logistic classifier model implies finding estimates of w that maximize the likelihood of the model (that is, the probability of the data given the model). It can be shown that this model is optimal when both the classconditional densities 𝑝(𝐱|𝐺) and 𝑝(𝐱|𝐵) are multi-normal with equal covariance matrices, where 𝐺 and 𝐵 represent the two target classes. The hyper-plane of all points 𝐱 satisfying the equation 𝐰T𝐱=0 forms the decision boundary between the two classes; these are the points for which 𝑝(𝐺|𝐱,𝐰)= 𝑝(𝐵|𝐱,𝐰)=0.5. 2.2.4. Generalized additive models Generalized additive models (GAM) are an extension on GLM, introduced by Hastie and Tibshrirani (1986). GAM is often integrated in commercial analytical tools that are used in scorecard designs (FICO, 2006). GAM is a family of powerful and palatable predictive modelling with a wide applicability range, suitable for business applications like credit scoring. It has been documented as a good solution to fulfill the gap between parametric and non-parametric predictive modelling, providing a good trade-off between 21 interpretability and predictive power (Silva and Cardoso, 2015). GAM generalizes the GLM procedure by replacing the linear predictor with a more general version 𝑠0+∑𝑠𝑗(𝑥𝑗) 𝑝 𝑗=1 , where 𝑠𝑗(.) are smooth functions standardized to verify 𝐸(𝑠𝑗(𝑥𝑗))=0 for 𝑗=1,..,𝑝 and 𝑝 representing the number of predictors. The 𝑠𝑗(.) function can be specified parametrically or not. 2.3. Critiques to the existing approaches and new challenges A few limitations to the existing approaches, idealized in the classical supervised classification paradigm, can be traced in published literature (Crook et al., 1992, Gama et al., 2014, Hand, 2006, Lucas, 2004, Thomas, 2010, Yang, 2007):  The static models usually fail to adapt when population changes.  Static and predefined sample settings often lead to an incomplete examination of the dynamics influencing the problem.  Certain assumptions that are implicit to the methods, often fail in realworld environments. These assumptions regard to: - Representativeness – the standard credit scoring models rely on supervised classification methods that run on 2-years-old static samples, in order to determine which individuals are likely to default in a future fixed period, 1 year for PD models. Such samples are supposed to be representative of the potential credit consumers of the future, the through-the-door population, and sufficiently diverse to reflect different types of repayment behaviour and to 28 A framework connected with this idea consists in collecting the new incoming data for sequential batches in predefined time intervals, e.g. year by year, month by month, or every day. The accumulation of these batches generates a panel data flow for dynamic modelling. In Finance, it remains unclear whether it is best having a long memory or forgetting old events. If on the one hand, a long memory is desirable because it allows recalling a wider range of adjust to the present situation. A rapid adaptation to changes is achieved with a short window, because it reflects the current distribution of default more accurately. However, for the contrary reason, the performance of models built upon shorter windows worsens in stable periods. In credit risk assessment modelling, this matter has been indirectly discussed by practitioners and researchers when trying to figure the pros and cons of using a through-thecycle (TTC) or point-in-time (PIT) schema to calibrate the output of the scorecards to the current phase of the economic cycle. For years, a PIT schema was the only option, because banks did not have sufficient historical data series. Since the implementation of the Basel II Accord worldwide, banks are required to store the data of default for a minimum 7-years period and consider a minimum of 5-years period for calibrating the scorecards. An original idea of Widmer and Kubat (1996) uses a sliding window of fixed length with a data processing structure first-in-first-out (FIFO). Each window may consist of a single or multiple sequential batches, instead of single instances. At each new time step, the model is updated following two processes. In the first process, the model is rebuilt based on the training dataset of the most recent window. Then, a forgetting process discards the data that move out of the fixed-length window. Incremental algorithms (Widmer and Kubat, 1996) are a less extreme hybrid approach that allows updating the prediction of models to the new contexts. 29 They are able to process examples batch-by-batch, or one-by-one, and update the prediction model after each batch, or after each example. Incremental models may rely on random previous examples or in representative selected sets of examples, called incremental algorithms with partial memory (Maloof and Michalski, 2004). The challenge is to select an appropriate window size. 2.5. Measuring performance Predictive modelling tries to find good rules (models) for guessing (predicting) the values of one or more variables in a dataset (target) from the values of other variables in the dataset. Our target is the quality of the individual, which can assume two values: bad or good. More than discriminating between these two possibilities, we will be interested in predicting the probability of defaulting (i.e. being bad). The models to use will then yield a scored dataset as a result of their training. The construction (training) of a model can be optimized to estimate only the probabilities of each class of the target variable, without incorporating any business objectives for which the predictor will be used. In our case the model would try to predict the probability of (not) default (i.e. being good). Next, we would be left with the decision of selecting a score cut-off, where individuals with risk score greater than or equal to a threshold would be accepted; others, below this cut-off would be rejected. This second stage would need to incorporate the adopted measure of business performance, be it profit, loss, volume of acquisitions, market share, etc. The ROC curves are well-suited for this second operation, enabling both to select the best cut-off for a given model and to compare multiple models, detecting dominant 30 models, points of intersection, etc. ROC curves are able to provide a richer measure of classification performance than scalar measures such as accuracy, error rate or error cost. Because they de-couple classifier performance from class skew and error costs, they have advantages over other evaluation measures such as precision-recall graphs and lift curves (Fawcett, 2006). However, as with any evaluation metric, there are some common misconceptions and pitfalls when using them in practice, so using them wisely requires knowing their characteristics and limitations (Hand and Anagnostopoulos, 2013). A two-step strategy has the benefit of being flexible in regards to changes in the measure of business performance. To accommodate a change in the loss or profit value, only the cut-off needs to be redefined, while the model is kept unchanged. Moreover, after selecting the best working point (cut-off), it is possible to perform a sensibility analysis, investigating how changes in the performance measure affect the performance of the model. It is important to stress that, often, losses or profits cannot be estimated with great certainty by experts on the company. Therefore, this two-step strategy is easily adapted to future changes. However, caution is in order: if the adopted measure of business performance leads to heavily different losses between the different possible errors, the operating point of the model will be strongly shifted away from the point for which it was trained (equal losses), possibly leading to a substantial degradation in performance. Another strategy would be to incorporate in the construction of the model the adopted measure of business performance. The training of the model would be focused not in the minimization of the misclassification rate but in the optimization of the profit or loss. In this case, the second stage of optimal cut- 31 off selection is (almost) unnecessary 8 . By integrating the business performance in the model construction we expect to attain an ‘optimal’ classifier, tuned for the business criterion. 2.5.1. ROC curves and optimal cut-off selection When designing a classifier we are essentially trying to minimize two types of errors: the error committed in identifying someone as defaulter, class B, when one is in fact a non-defaulter, class G, individual and the opposite type of error of diagnosing someone as non-defaulter when one is in fact a defaulter. A confusion matrix (Table 2.2) can be used to lay out the different errors: Table 2.2: Confusion matrix C. Predicted class True Class Defaulter, 𝐵  Non-defaulter, 𝐺  Defaulter, B 𝑝(𝐵,𝐵 ) 𝑝(𝐵,𝐺 ) Non-defaulter, G 𝑝(𝐺,𝐵 ) 𝑝(𝐺,𝐺 ) In this confusion matrix, 𝑝(𝐺,𝐵 ) represents the probability of that model predicts as defaulter and the real class is non-defaulter; the other probabilities follow accordingly. Note that 𝑝(𝐵,𝐵 )+𝑝(𝐵,𝐺 )=𝑝(𝐵), the a priori default 8 However, not all models allow the incorporation of the loss or profit matrix in the construction process; others use it in a simplified or approximated mode. Therefore, a second stage of tuning the cut-off may reveal appropriate. 32 probability in the population. Likewise, 𝑝(𝐺,𝐵 )+𝑝(𝐺,𝐺 )=𝑝(𝐺), the a priori non-default probability in the population and naturally 𝑝(𝐵)+𝑝(𝐺)= 1. We would like to minimize both 𝑝(𝐵,𝐺 ) and 𝑝(𝐺,𝐵 ). At one extreme case if our classifier predicts defaulter for any individual we would have 𝑝(𝐵,𝐺 )= 0, but a presumably high 𝑝(𝐺,𝐵 ) (in fact, equal to 𝑝(𝐺)); at the other end if the trained classifier predicts always non-defaulter we would have 𝑝(𝐺,𝐵)= 0, but a non-zero 𝑝(𝐵,𝐺 ) (in fact, equal to 𝑝(𝐵)). So, designing a classifier resumes to finding the best trade-off between these two types of errors. And the best trade off depends on the costs associated with each decision. Consider a generic loss matrix, LM (Table 2.3). Table 2.3: Generic loss matrix LM. Predicted class True Class Defaulter Non-defaulter Defaulter 𝑙1 𝑙2 Non-defaulter 𝑙3 𝑙4 It is easy to see that the expected loss, 𝐸[𝐿], for a classifier with the confusion matrix C is: 𝐸[𝐿]=𝑙1×𝑝(𝐵,𝐵 )+𝑙2𝑝(𝐵,𝐺 )+𝑙3×𝑝(𝐺,𝐵 )+𝑙4𝑝(𝐺,𝐺 ) (5) Now 𝑙1×𝑝(𝐵,𝐵 )+𝑙2𝑝(𝐵,𝐺 ) 33 =𝑝(𝐵)[𝑙1×𝑝(𝐵,𝐵 ) 𝑝(𝐵)+𝑙2×(1−𝑝(𝐵,𝐵 ) 𝑝(𝐵) )] =𝑝(𝐵)[(𝑙1−𝑙2)×𝑝(𝐵,𝐵 ) 𝑃(𝐵) +𝑙2] (6) where 𝑝(𝐵,𝐵 ) 𝑝(𝐵)=𝑝(𝐵 |𝐵) is usually known as sensitivity or true positive rate. In the same way 𝑙3×𝑝(𝐺,𝐵 )+𝑙4×𝑝(𝐺,𝐺 ) =𝑝(𝐺)[(𝑙3−𝑙4)×𝑝(𝐺,𝐵 ) 𝑝(𝐺) +𝑙4] =𝑝(𝐺)(𝑙3−𝑙4)(1−𝑝(𝐺,𝐺 ) 𝑝(𝐺))+𝑝(𝐺)×𝑙4 (7) where 𝑝(𝐺,𝐺 ) 𝑝(𝐺) =𝑝(𝐺 |𝐺) is usually known as true negative rate or specificity. Then, 𝐸[𝐿] can be assumed as 𝑝(𝐵)(𝑙1−𝑙2)×sensitivity+𝑙2𝑝(𝐵)+𝑃(𝐺)(𝑙3−𝑙4)(1−specificity)+𝑙4𝑝(𝐺) (8) Summarizing, the loss of a classifier depends linearly only on two parameters, the sensitivity and specificity, weighted by coefficients derived from the loss matrix and the a priori class probability on the population. This means that we can analyse the performance of a model in a 2D space, the (1specificity, sensitivity) space. However, not all points in this space are possible for a model. Having trained a classifier to output the probability of defaulting, we can vary the cut-off parameter (the probability value at which we start declaring a client as defaulter) and, as we do so, trade specificity by sensitivity. Fig. 2.1 illustrates 34 an example of an ROC chart for two different classifiers. The ROC of a classifier shows this trade-off, plotting the achievable (1-specificity, sensitivity) values for a range of cut-offs. Fig. 2.1: ROC chart for two different classifiers. Each point on the curve represents a cut-off probability. Points closer to the upper-right corner correspond to low cut-off probabilities. Points in the lower left correspond to higher cut-off probabilities. The extreme points (1,1) and (0,0) represent no-data rules where all cases are classified into class Defaulter or class non-defaulter, respectively. Now, as shown above, the expected loss can be represented as a linear function of sensitivity and 1-specificity: 𝐸[𝐿]=𝑎×(1−specificity)+𝑏×sensitivity+𝑐. (9) To minimize the loss we just have to walk in the direction opposite to the gradient of 𝐸[𝐿]. It is not difficult to see that the represented classifier B is dominant over classifier A, in the sense that, for any (reasonable) loss matrix considered, there is always an operation point of classifier B that is better than the best operating point of classifier A. 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1-specificity sensitivity classifier A classifier B 35 A different situation is depicted in Fig. 2.2. The ROC curve of classifier A intersects the ROC curve of the classifier C. Now, the best classifier depends on the loss matrix. The isocost line (a isocost line is a line of constant cost, perpendicular to the gradient of the loss function) shows the least loss line for a loss matrix M2 where the costs of missing a positive case severely outweighs the cost of raising a false alarm; in this case classifier C provides the best operating point. Conversely, the dotted isocost line shows the least loss line for a loss matrix M1 where the costs of missing a negative case severely outweighs the cost of raising a positive alarm. Now is classifier A that provides the best operating point as illustrated in Fig. 2.2. Fig. 2.2: Non-dominant ROCs. 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1-specificity sensitivity classifier A classifier C gradient for a Loss Matrix M1 gradient for a Loss Matrix M2 iso-cost line for Loss Matrix M2 iso-cost line for Loss Matrix M1 36 2.5.2. Comparing models performance Throughout this research the experimental comparison of different models (or methods) is based in their ability to discriminate between the two target classes. The discriminatory power is measured with the Gini coefficient, a typical evaluation criteria among researchers and in the industry (Rêzác and Rêzác, 2011), which can be calculated as 2×𝐴𝑈𝐶−1, where AUC is the area under the ROC curve. The Gini coefficient refers to the global quality of the credit scoring model, and may range between -1 and 1. The perfect scoring model fully distinguishes the two target classes, good and bad, and has a Gini index equal to 1. A model with a random output has a Gini coefficient equal to zero. If the coefficient is negative, then scores have a reverse meaning. The extreme case -1 would mean that all examples of the good class are being predicted as bad, and vice-versa. In this case, the perfect model can be achieved just by switching the prediction. Loans’ records with unknown score are not included in the calculations of this indicator. 37 3. Credit scoring models degradation Abstract - Evidence from the Freddie Mac’s single loan-level dataset, first published in March 2013, shows that existing scores are effective to order individuals by risk, but they are not prepared to predict real default in each point in time. We investigate the dynamics and performance of over 16.7 million of fully amortized 30-year fixed-rate mortgages in the U.S., originated between 1999 and the first quarter of 2013. We identify the frailties of the frameworks used in default prediction, to draw implications to risk-based pricing designs. Analysis shows that not only scores diminished their ability to predict default when the mortgage crisis has come to public’s attention, but also that real default rates by score are irregular over time. It is also apparent that, since 2009, lenders are firmly declining the subprime loans, and as a result the 1year cumulative default by vintage has declined. There is a link between scores, lending and default, mostly influenced by the lending practices. There is a link between scores, default and pricing, but the mapping between them is far from being adequate. Parts of this chapter are published in the Journal of Economics, Business and Management with the title “Links between Scores, Real Default and Pricing: Evidence from the Freddie Mac’s Loan-level Dataset”. JOEBM 2015 Vol.3(12): 1106-1114 ISSN: 2301-3567; DOI:10.7763/JOEBM.2015.V3.343. Some sections are repeated through the dissertation to make the chapters self-contained. 44 of lending decisions, to determine how much money each individual can borrow, and how much interest he will pay. In the OECD countries, banks that have adopted the Internal Rating Based Approach (IRB) in Basel II, internally developed credit scoring models play an essential role in the calculation of the minimum regulatory capital. In the European Market, there are 89 banks using the IRB. In the U.S., the largest banks also adopted the Basel II Accord, introduced via Capital Requirements Directive. In line with this evolution, financial industry moves toward a more intensive use of credit scores at the basis of risk-based pricing models. 3.3.2. Scoring models – strengths and frailties The huge success of credit scoring models in the advanced economies is partly explained by their appealing representation is a linear scale. Credit scoring models have also proved to a powerful measure to order a population of individuals according to their credit risk. The best scoring model is the one that differentiates the most the two target classes, good and bad, commonly referred as discriminatory power. 45 3.3.2.1. Human misconception Although the true meaning of scores is a probability of default (PD) with a non-linear shape, as in Fig. 3.1(a), human cognition retains the linear representation, Fig. 3.1(b), rather than the actual non-linear shape. Through our experience in developing credit scoring models it became apparent that many of the risk managers were basing their credit risk assessments on the linear representation. Doing so is suitable for ordering risks, but it is insufficient to calculate losses or pricing credit risks. (a) True meaning - Probability of default by score. (b) Misconception - Human cognition of risk by score. Fig. 3.1. Actual meaning of scores and human misconception of the risk. Authors’ analysis, illustrative scorecard: the score 660 has a good/bad odd of 15 good individuals to 1 bad, and for each additional 15 score points the good/bad odds double. 3.3.2.2. Economic cycle and scores misalignment Traditional systems that are the basis of credit scoring models are one-shot, fixed memory-based, trained from fixed training sets. Since static models are 0.0 0.5 1.0 300 345 390 435 480 525 570 615 660 705 750 795 840 Probabily of default Score 0.0 0.5 1.0 300 345 390 435 480 525 570 615 660 705 750 795 840 Risk degree Score 46 not prepared to process the highly detailed evolving data, they are not able to continuously maintain an output (PD) consistent with the current state, or to quickly react to changes (Gama, 2010). When there are significant changes in the conditions, scores’ PD (Fig. 3.2, dashed line) may become misaligned 2 with the observed default (Fig. 3.2, solid line). Fig. 3.2. Illustration of scores misalignment. Authors’ analysis based on the Freddie Mac’s database. Forecasted PD’s by score same as the default rate of the mortgages originated in 1999, and observed PD’s the real default rates in 2007. Performance measured in the 1-year after the loans were originated. As the processes underlying credit risk are not strictly stationary, consumers’ behaviour and default can change over time in unpredictable ways. There are several types of evolution inside a population, like population drifts, that translate into changes in the distributions of the variables, affecting the performance of the models. When the economic conditions evolve in the economic cycle, deteriorating or improving, the behaviour of the individuals, and their ability of repaying their debts also vary. In addition, default evolution echoes trends of the business 2 The terms alignment, adjustment and calibration are commonly used with the same meaning. Observed Forecasted 0% 2% 4% 6% 8% 10% 12% Default Score 47 cycle, and related with this, regulatory movements, and interest rates fluctuations. In good times, banks and borrowers tend to be overoptimistic about the future, whilst in times of recession banks are swamped with defaulted loans, high provisions, and capital buffers turn highly conservative. The former leads to more liberal credit policies and lower credit standards, the later promotes sudden credit-cuts. Empirical evidence and theoretical frameworks support a positive, and lagged relationship between rapid credit growth and loan losses (Sousa et al., 2015a). In order to adapt models’ output to changes over time, institutions should calibrate their scoring models according to the most recent information. Models’ adjustments, or calibration, commonly consider selected macroeconomic public indicators and should be periodically revised. However, this may take too long to occur. The European Banking Authority reports that there is not a common practice among Regulators towards models calibration. Many countries do not define any specific rules and, when they do, they are usually not public. When they define some rules, they are rarely convergent; and different countries favour different calibration choices (EBA, 2013b). Should there be significant changes in between scheduled modelling developments or adjustments, it is not certain that banks will anticipate any of these tasks, as it can largely depend on judgmental reasoning or overlayered decision frameworks. In the following, we will analyse the Freddie Mac’s database to find evidence on the previous limitations. Complementing the previous works, we provide a fine-graded time-analysis over scores and analyse in which conditions riskbased pricing has been implemented. Furthermore, we study the extent of misalignment of PD’s with the real default by score over time. 48 3.4. Experimental design The research summarized here was conducted in the Freddie Mac’s single family mortgage loan-level dataset, first published in March 2013. We follow the performance of 16.737 million of fully amortized 30-year fixed-rate mortgages loans in the U.S., originated between January 1, 1999 and March 31, 2013. The loans performance 3 is outlined in a monthly basis and, at the time of this research, data for performing loans and those that were up to 180 days delinquent were available through September 30, 2013. The dataset is a “living” dataset updated over time, typically at the end of each quarter, and may be subjected to periodical corrections by Freddie Mac. The release changes are recorded (Freddie Mac, June 2014a). A general user guide describing the file layout and data dictionary is also provided (Freddie Mac, June 2013b). Freddie Mac’s information regarding the key loan attributes and performance metrics can be linked to our research in the aggregated summary statistics. 3 Loan performance information includes the monthly loan balance, delinquency status and information regarding termination events: Voluntary prepayments in full; 180 days delinquency (“D180”); Repurchases prior to D180; Third-party sales prior to D180; Short sales prior to D180; Deeds-in-lieu of foreclosure prior to D180; Real estate owned (REO) acquisition prior to D180. Also includes voluntary prepayments and loans that were short sales, deeds-in-lieu of foreclosure, third party sales, and REOs. 49 3.4.1. Methodology We attempt to describe the most significant events in the period. We are both interested in determining the main contrasts by score, and to illustrate the dynamics over time. First, we illustrate the volumes and compare the original interest rate with the annual average FIX 30 4 . In so doing, our aim is to use a representation of the credit risk spread evaluation over time and understand the extent of under-priced loans that has been referred as one cause of the crisis. Then, we determine the evolution of default over time and the performance of the scores at the basis of the credit risk assessment. For assigning the default event, we used the information of the loan delinquency status in each reporting period. In this analysis we consider that a borrower entered in default if he was ever 90 or more days delinquent, the typical definition used under the Basel II. Default is assigned to the first occurrence of this event. We use vintage analysis and hence, we consider cumulative default along time. 3.4.2. Data aggregation Although we have found missing values in the score, we intentionally kept these cases in the analysis to entirely represent the extent of information (or absence) at the basis of the original risk assessment. Data of the original datasets were aggregated by the origination year. 4 FIX 30 is the interest rate of a 30-year fixed-rate mortgage. 50 Scores may vary in the range 301-850, or be unknown. Situations where the score is unknown are described by Freddie Mac (June 2013b) 5 . To compare evolutions over time, we divided the range into equidistant intervals of 25, except for the lower and upper bounds. To have dimension, these bounds were aggregated in the buckets [300, 550[ and [800, 850[, respectively. 3.4.3. Scores - performance, concentration and stability measures The discriminatory power of the model was measured based on the Gini coefficient, equivalent to consider the area under the ROC curve (AUC), which is a typical evaluation criteria among researchers and in the industry (Rêzác and Rêzác, 2011). This coefficient refers to the global quality of the credit scoring model, and may range between -1 and 1. The perfect scoring model fully distinguishes the two target classes, good and bad, and has a Gini index equal to 1. A model with a random output has a Gini coefficient equal to zero. If the coefficient is negative, then the scores have a reverse meaning. The extreme case -1 would mean that all examples of the good class are being predicted as bad, and vice-versa. In this case, the perfect model can be achieved just by switching the prediction. Loans’ records with unknown score were not included in the calculations of this indicator. The concentration of loans by score buckets was measured with the Herfindahl-Hirschman Index (HHI), which is defined as ∑𝑓𝑖2 𝑛 𝑖=1 , where n is the number of score buckets and 𝑓𝑖 is the number of customers in that bucket 5 A possible reason is when the seller requires a reduced level of verification. 51 relative to the total portfolio. By definition, the index varies between 0 and 100%. An index of 100% means that borrowers are concentrated in a single bucket. In this work we will consider that values below 20% are commonly acceptable. Values above it suggests highly concentrated scores. The stability index was measured comparing the distribution of the population in each year with the distribution of the population in the first year in the period, 1999. For the year 𝑡, 𝑡=2000,..,2013 the stability index is calculated as: ∑(𝑓𝑖,1999−𝑓𝑖,𝑡)∗𝑙𝑛(𝑓𝑖,1999 𝑓𝑖,𝑡 ⁄) 𝑛 𝑖=1 (1) where n is the number of score buckets, and 𝑓𝑖,𝑡 is the number of borrowers in that bucket relative to the number of borrowers in the total portfolio in the year t. 3.5. Results There are a number of theories regarding the origins of the mortgage crisis. In this chapter we are concerned to extend knowledge on the potential misalignment of the risk indicators that were at the basis of credit approval before crisis. Then, we assess the extent of undervaluation of the credit risk, and hence, credit risk mispricing. As credit risk assessment is anchored in the borrowers’ score at the origin of the loan, the analysis is focused in the dimensions score and time. Results are mostly motivated to present evidence on the following: 52  Risk taking and pricing;  Changes in the lending practices after the crisis;  Risk assessment over time - default rates, default misalignment and scores performance. 3.5.1. Risk taking and pricing The evolution of new loans over the analysed period illustrates the U.S. housing bubble between 2001 and 2005. The highest peak occur between 2001 and 2003, where the number of new loans continuously rose from nearly 800 thousand loans in 2000 to 1,930 thousands in 2003 (Table 3.1, first row). This massive increase of the number of loans was one of the sources of the raise in the real state property values, which reached a peak in 2005. Our analysis confirms that scores are intensively used to differentiate the mortgages interest rates. As illustrated in Table 3.2, there is a decreasing trend of the average interest rate from the lower to the higher score buckets. It can be said that a risk-based pricing based in scores is being applied. Until 2009, borrowers with the highest scores were borrowing below the average FIX 30. In 2007, the default of the borrowers with scores 650 or higher has almost tripled in relation to the previous years. Adjustments to the risk premium were made by 2009, as it can be seen in Fig. 3.3. From this point onwards, rates are higher than FIX 30, suggesting that pricing policy had been revised. Loans have been priced below the FIX 30 in 2001, both in the aggregate level and in each score bucket (Table 3.1, rows 7, 8 and 9). Loans’ average rate was maintained through 2001 and 2002, in the aggregate level and in score buckets’ level (Table 3.2). This effect may be linked to the crash of the dot- 53 com bubble in 2000 which has been associated to the beginning of the decline in real long-term interest rates. In reaction to the crash of the dot-com bubble in 2000 and to the recession that began in 2001, the Federal Reserve Board has cut short-term interest rates from 6.5% to 1% (Bianco, 2008). Mortgage interest rates continued to decline until 2005 (Table 3.2). As the mortgage rates are typically set relation to 10-year Treasury bond yields, this was an outcome of very low Fed funds’ rates in the period. Lenders were self-reliant that they were taking little risk because the value of the collateral was rising too fast, but they missed to understand that it would come to an end. Loans underwritten between 2001 and 2005 account for 42% of the amount originated in the period (Table 3.4). There is a theory (Hull, 2009) referring that in this period, lenders had begun to take more risk in subprime 6 mortgages. Our analysis provides a divergent finding, because neither the number of loans nor the amount granted has increased during the mortgage bubble (Table 3.3 and Table 3.4, see rows for scores’ buckets bellow score 625). 6 A rule of thumb for the subprime mortgage is a loan of a borrower with a score inferior to 620. Some lenders also consider a subprime mortgage if the borrower has a score up to 680 and the down payment is less than 5% of the loan. 60 models alignment, either when they are used in credit decision making or in risk-based pricing designs. 3.6. Conclusions Financial industry turned over-dependent on credit scoring in the advanced economies. A high proportion of the loan applications are automatically decided. In this framework, credit score is the central, if not the unique, indicator of the borrowers’ credit risk. We found evidence that the cumulative default by vintage has almost tripled in first years of crisis for scores equal to 650 or higher, suggesting that credit risk may has been under-priced in these cases. Two years after the crash, lending decision threshold changed and lending moved markedly to borrowers with higher scores, which led to an increase in concentration of lending in these individuals. Although this is a reasonable prudential measure, excessive lending bias and concentration towards the highest scores require more precise default estimation to correctly price credit risk. Therefore, credit scoring models should properly adapt to time-changing conditions and lending dynamics, so that they faithfully support risk taking and pricing. Any misalignment between the PD’s by score and the real default over time will guide to inconsistent decisions and suboptimal prices. There is a new emphasis on running predictive models with the ability of sensing themselves and learn adaptively (Gama, 2010, Adams et al., 2010, Pavlidis et al., 2012). This is one area where more sophistication is needed and more effort should be put to promote their wider acceptance. 61 4. A first approach to dynamic credit scoring Abstract - In this chapter we propose a two-stage model for dealing with the temporal degradation of credit scoring models. First, we develop a model from a classical framework, with a static supervised learning setting and binary output. Then, we introduce the time-changing economic factors, using a regression between the macroeconomic data and the internal default in the portfolio. In so doing, the specific risk is captured from the bank internal database, and the movement of systemic risk is determined with the regression. This methodology produced motivating results in a 1-year horizon, for a portfolio of customers with credit cards in a financial institution operating in Brazil. We anticipate that it can be extended to other applications of risk assessment with great success. This methodology can be further improved if more information about the economic cycles is integrated in the forecasting of default. Parts of this chapter have been published in the proceedings of the conference BRICS Countries Congress on Computational Intelligence (BRICS-CCI), held in Brazil in 2013, with the title “A two-stage model for dealing with temporal degradation of credit scoring”. The methodology was used in the model than won the competition in the area of data mining and Finance promoted by that conference. Some sections are repeated through the dissertation to make the chapters self-contained. 62 4.1. Introduction In retail banking, credit risk assessment often relies in credit scoring models developed with supervised learning methods used to evaluate a person’s credit worthiness, so-called scoring or PD models 1 . The output of these models is a score that translates a probability of a customer becoming a defaulter, usually in a fixed future period. Nowadays, these models are at the core of the banking business, because they are central to credit decision-making, in price settlement, and to determine the cost of capital. Moreover, central banks and international regulation have rapidly evolved to a structure where the use of these models is implicit, to achieve soundness standards for credit risk valuation in banking system. Since 2004, with the implementation of regulations issued by the Basel Committee on Banking Supervision within Basel II, banks were encouraged to strengthen their internal models frameworks for reaching the A-IRB (Advanced Internal Rating Based) accreditation. To achieve this certification, banks had to demonstrate that they were capable of accurately evaluating their risks, complying with Basel II requirements, by using their internal risk models’ systems, and keep their soundness. Banks owning A-IRB accreditation gained an advantage over the others, because they were allowed to use lower coefficients to weight the exposure of credit at risk, and benefit from lower capital requirements. A lot of improvements have been made in the existing rating frameworks, extending the use of data mining tools and artificial intelligence. Yet, this may have been bounded by a certain 1 Other names can be used to refer to PD models, namely: credit scoring, credit risk models, scorecards, credit scorecards, rating systems or rating models, although some have different meanings. 63 unwillingness to accept less intuitive algorithms or models going beyond standard solutions being implemented in the banking industry, settled in-house or delivered through analytics providers (e.g. FICO, Experian, PwC and KPMG). To our knowledge, developing and implementing a credit scoring model can be time and resources consuming – easily ranging from 9 to 18 months, from data extraction until deployment. Hence, banks use unchanged credit scoring models for several years. Bearing in mind that models are built using a sample file frequently comprising 2 or more years of historical data, in the best case scenario, data used in the models training are shifted 3 years away from the point they will be used. However, to our knowledge an 8-year shift is frequently exceeded. Should conditions remain unchanged, then this would not significantly affect the accuracy of the model, otherwise, its performance can greatly deteriorate over time. The recent financial crisis confirmed that financial environment significantly fluctuates unexpectedly, bringing renewed attention regarding scorecards built upon frames that are by far outdated. By 2007-2008, many financial institutions were using stale scorecards built with historical data of the early-decade. The degradation of stationary credit scoring models is an issue with empirical evidence in the literature (Crook et al., 1992, Lucas, 2004), however research is still lacking more realistic solutions. Dominant approaches usually stand on static learning models. However, as the economic conditions evolve in the economic cycle, deteriorating or improving, the behaviour of the individuals also vary and their ability of repaying their debts. Furthermore, default evolution echoes trends of the business cycle, and related with this, regulatory movements, and interest rates fluctuations. In good times, banks and borrowers tend to be overoptimistic about the future, whilst in times of recession banks are swamped with 64 defaulted loans, high provisions, and tighten capital buffers. The former leads to more liberal credit policies and lower credit standards, the later promotes sudden credit-cuts. Empirical evidence and theoretical frameworks support a positive, and lagged relationship between rapid credit growth and loan losses (Sousa et al., 2015a). Therefore, default needs to be regarded as time changing. Public studies are mostly motivated by capital consumption and regulatory concerns, and therefore they are focused in modelling macroeconomic factors. So far, none has explicitly integrated these factors in the existing credit scoring models. Traditional systems that are one-shot, fixed memory-based, trained from fixed training sets are not prepared to process the highly detailed evolving data. Therefore, they are not able to continuously maintain an output model consistent with the actual state, or to quickly react to changes (Gama, 2010). These are some of the features of classic approaches that show that the existing credit scoring systems are limited. As the processes underlying credit risk are not strictly stationary, consumers’ behaviour and default can change over time in unpredictable ways. There are several types of evolution inside a population, like population drifts, that translate into changes in the distributions of the variables, affecting the performance of the models. There is a new emphasis on running predictive models with the ability of sensing themselves and learn adaptively (Gama, 2010). Advances on the concepts for knowledge discovery from data streams suggest alternative perspectives to identify, understand and efficiently manage dynamics of behaviour in consumer credit in changing environments. In a world in which events are not preordained and little is certain, what we do in the present affects how events unfold in unexpected ways. 65 This chapter follows in section 4.2 with a brief description of the problem and the research objectives; in section 4.3 the database is succinctly presented. Section 4.4 details the methodology and theoretical framework of this chapter. A one-dimensional analysis is presented, as well as an overall assessment of the data available for modelling. A multidimensional approach is exposed in section 4.5 where we propose a two-stage model for dealing with the temporal degradation of credit scoring. In the first stage we develop a credit scoring, by comparing several supervised learning methods. In the second stage, we introduce the effect of time changing environment by shifting the initial predictions by a factor of the expected variation of default. Results in a 1-year horizon are presented in section 4.6. Conclusions and future applications of the two-stage model are discussed in section 4.7. 4.2. Problem and research objectives The aim of this research is to propose a new approach for dealing with the temporal degradation of a portfolio of customers with credit cards in a financial institution operating in Brazil. Our work is attached to the BRICS 2013 competition, and is based in a real world dataset, along two years of operation, from 2009 to 2010. This competition consisted of two tasks, each focused on two features of the credit risk assessment model: Task 1: Develop a scorecard, tilting between the robustness in a static modelling sample and the performance degradation over time, potentially caused by market gradual changes along few years of business operation. 66 Task 2: Fitting of the estimated delinquency produced by an estimation model to that observed on the actual data for the applications approved by the scorecard. Participants were encouraged to use any modelling technique, under a temporal degradation or concept drift perspective. The official performance metrics were the area under the ROC curve (AUC) for task 1, and the Chi-square for the monthly estimates of delinquency in task 2. Innovative ways of handling task 1 can be found in PAKDD 2009 Competition whose focus was on this type of degradation. Task 2 represents an innovation in data mining competitions worldwide by emphasizing the quality of future delinquency estimation instead of the usual lowest future average delinquency. We have built an integrated solution to deal with these two tasks. First we developed a credit scoring using a set of supervised learning methods. Then we calibrated the output, based on a projection of the evolution in the default. This forecast considered both the evolution of default and the evolution of exogenous data series, echoing potential changes in the population of the model, in the economy, or in the market. In so doing, resulting adjusted scores translate a combination of the customers’ specific risk with systemic risk. As in many other applied financial studies, this research is bounded by some practical limitations, like systematic noise in the data and short time series. Our view is that the choice of the most appropriate methods for developing a credit scoring is context specific, and therefore, we present a technical approach driven by the specifics of the problem. We have also taken in consideration the extent of meaningful and reliable data that is actually available for modelling. 67 4.3. Database and development environment The research summarized in this chapter was conducted in a real-life financial dataset, comprising 762,966 applications of credit cards, from a financial institution in Brazil. Data for modelling was provided along two years of operation, from 2009 to 2010. Each customer in the modelling dataset is assigned to a delinquency outcome - good or bad. In this problem, a person is assigned to the bad class if she had a payment in delay for 60 or more days, along the first year after the credit has been granted. The delinquency rate in the modelling dataset is 27.3%. Two additional datasets from the next year, 2011, were used to test the performance achieved in the static modelling sample, namely the leaderboard and the prediction dataset. The leaderboard contains a sample of 60,000 records that were obtained aggregating subsamples of 5,000 applications of each month in 2011. Although the default outcome was not available in the leaderboard dataset, the discriminatory power of the model given by the receiver operating characteristic curve (AUC) could be measured during the modelling stage. Competitors were allowed to make submissions of their solutions in the leaderboard, and for each the AUC and the distance D were measured online. The prediction dataset was used for the final performance evaluation in the competition. This dataset has 444,828 applications in 2011, for which the default outcome was not available at any stage of the modelling. A summary of the files is presented in Table 4.1. 68 Table 4.1: Case study Brazil - dataset summary. Dataset Records Period Target Delinquency (%) Modelling 762,966 2009-2010 Labelled 0.273 Leaderbord 60,000 2011 Unlabelled Unknown Prediction 444,828 2011 Unlabelled Unknown The full list of variables in the original dataset was downloaded from the BRICS 2013 official website. The dataset contains 39 variables, summarized in Table 4.2 and one binary target variable with values 1 identifying a record in the bad class and 0 for the good class. Table 4.2: Case study Brazil - variables summary. Type # Information Numerical 6 Age, monthly income, time at current address, time at current employer, number of dependents, and number of accounts in the bank. Treated as nominal 13 Credit card bills due date, 1st to 4th zip digit codes, home (state, city, and neighbourhood), marital status, income proof type, long distance dialling code, occupation code, and type of home. Binary 16 Address type proof, information of the mother and father’s names, input from credit bureau, phone number, bills at the home address, previous credit experience, other credit cards, tax payer and national id, messaging phone number, immediate purchase, overdraft protection agreement, lives and work in the same state, lives and work in the same city, and gender. Date 1 Application date. ID 3 Customer, personal reference, and branch unique identifiers. 69 4.4. Methodology and theoretical framework This research evolves from a one-dimensional analysis, where we come across the financial outlook underlying the problem, to a multidimensional approach, where we gradually develop and experiment a new framework to model credit risk. The one-dimensional analysis was tailored to gain intuition on the default predictors and the main factors ruling the dynamics of default. The multidimensional approach is at the core of the work presented in this chapter and was held in two stages. In the first stage we developed a credit scoring model from a classical framework, with a static learning setting and binary output. In the second stage, we used a linear regression between exogenous data (fully listed in Table 4.4) and the internal default for adjusting the predictions. In the first stage, we used the exogenous data series disclosed by the Central Bank of Brazil (Banco Central do Brasil, 2011) and the Brazilian Institute of Geography and Statistics 2 to evaluate the effect of time changing economics in the default evolution. We used quarterly series available from January 2004 through December 2011. Nevertheless, for determining the fitting between them and the internal default, only the period 2009 to 2010 was considered. We used the coefficient of determination, r-square, to evaluate the quality of fitting between the exogenous series and the default in the analysed portfolio, in a one-dimensional basis. 2 Source: www.tradingeconomics.com. 76 expenditure, and the inflation rate. As only 8 observations are available, linear regression should consider a single independent variable, aiming to avoid overfitting. In forecasting scenarios, we only considered the series with the highest correlation - default on the financial revenues of credit cards in Brazil. Notwithstanding, an r-square of 64% in the regression can be considered low for the regression. We tested three forecasting scenarios, summarized in Table 4.5, which were iteratively submitted to the leaderboard. The final prediction is based on the scenario with the lowest distance D in the leaderboard. Table 4.5: Case study Brazil - forecasting scenarios tested in task 2. # Description Rational 1 Estimate the default in each quarter adaptively from the values of default in credit cards of the previous quarter, and submit calculated values. Incorporate new information adaptively, when it is available. This may benefit from the drift detection and suggest implementing corrective actions. 2 Estimate the default in each quarter adaptively from the values of default in credit cards of the previous quarter, and submit the average. As there is no knowledge about the economic cycle, any correction resulting from a drift between quarters may be less apropriate in this application. It may be preferrable to use a central tendency of default (e.g.average defaults). 3 Submit the average annual default until September, and the average increased by 1% in the last two months of the year. Use central tendency of default and adjust the months were the direction of the drift is more certain. 77 4.5. Two-stage model Some previous research suggests including economic conditions directly into a regression scorecard (Zandi, 1998), survival analysis (Bellotti and Crook, 2009), or transition models (Malik and Thomas, 2012). Our approach was to use a two-stage modelling framework in order to keep separate the two dimensions of risk – specific and systemic. The specific risk should be captured from the bank internal database with the scorecard developed in the first stage, and the movement of systemic risk is calculated with a linear regression at a second stage. The final score is therefore an adjustment of the initial score by the expected variation of the default in the population of the model. 4.5.1. Credit scoring construction Several standard classification models, based on logistic regression (LR), AdaBoost, and Generalized Additive Models (GAM) were designed and tested with a 10 fold crossed-validation. Characteristics were iteratively added to the models, until no performance gain was observed in the test. Four different strategies were gauged, varying the window, and eliminating noise from the input dataset: High volume and diversity - Use the entire modelling dataset (20092010), for more volume and diversity. Through-the-door - Use the sample closest to the through-the-door population, for mitigating the effects of the temporal bias. We tested two different windows: 2010 full year, and 2010 last quarter. 78 Ensemble of 12 models - Create a model to apply in each month of the year, using the corresponding months in 2009 and 2011. Our idea is to address the changes in demand that may be attached to seasonal effects. Twelve models were developed, and their results were combined in the final score. Cleaning - To overcome the presence of noise in the data, create a model using the entire modelling dataset (2009-2010); since the presence of noise in the data can spoil the model, we removed the examples strongly misclassified, and retrained the model in the reduced set. We consider an example strongly misclassified if the predicted posterior probability of the true class is less than 0.05. As the financial institution is expanding, the leaderboard and the prediction sets should contain new codes in the discrete variables (e.g. new branch and geographical codes). These cases cannot be trained from the modelling dataset, because they cannot be observed before the expansion. As there was no knowledge about the nature and strategy of the expansion, we opted to assign an educated guess - use the average partial score for unfamiliar codes. In real-word environments, this assignment can be better guided, because the expansion strategy is known in advance. 4.5.2. Introducing the effect of time changing environment In the second stage of modelling we introduced the effect of time changing environment into the scorecard previously developed. The initial predictions of the scorecard were shifted according to a factor of the expected variation of default. Every cut-off point set between 0 and 1 is converted into a new 79 adjusted default rate within the set of approved applications, as illustrated in Fig. 4.2. Fig. 4.2: Case study Brazil – model adjustment with the default central tendency. 4.6. Results Task 1 - Only the models with the best results in the test set were submitted to the leaderboard. Table 4.6 shows the results for the best models in the modelling dataset. Although a broad number of different configurations were tested, we only present the results for the models with an AUC higher than 0.71 in the test set. This includes the models that were also submitted to the leaderboard. Sample Adjusted 0 10 20 30 0 1 Default rate (%) Score Cut-off Expected default 80 Table 4.6: Case study Brazil - best models in task 1, AUC>0.71 in the test set. Method Period Test Leaderboarda GAM 2009-10 0.7320 0.7227 GAM 2010 0.7140 0.7131 GAM 2010(Q4) 0.7204 n.a. LR 2009-10 0.7222 n.a. LR with cleaning 2009-10 0.7222 n.a. LR monthly 2009-10, month against month 0.7230 0.7140 AdaBoost 2009-10 0.7180 n.a. a. n.a. - not available, because the model was not submitted to the leaderboard. The model with the best results in the leaderboard is based on the Generalized Additive Model, and using the entire modelling dataset (2009 and 2010). This model was also submitted in the final prediction of the competition. Although we have tested different timeframes when modelling, it became apparent that for this application, the model with best performance was achieved with the larger volume of examples and diversity. The degradation of the proposed scorecard from the test (2009-2010) to the leaderboard (2011) is of 0.93%, which is quite acceptable in similar real-world applications. The degradation was partly controlled by adjusting unstable variables (e.g. monthly income based on the inflation rate). Apart from these, the bulk of remaining the characteristics is quite stable over time. Task 2 – The results of this task, presented in Table 4.7, demonstrate that when the forecast depend on very short time series, the best fitting is achieved with the simplest adjustment – the central tendency of default. In this problem, submitting an average default enhanced the fitting of default 81 (scenario 2), which was further improved by adjusting on the months where the direction of the drift is more certain (scenario 3). Table 4.7: Case study Brazil - results in task 2. Scenario (#) Average default rate Distance D 1 0.293 3.16 2 0.293 1.45 3 0.294 0.83 a. Annual average forecasted default rate in 2011, if the entire portfolio is approved. 4.7. Conclusions Theoretical models for knowledge extraction from data streams seem suitable for dealing with temporal degradation of credit scoring models. The idea is to use adaptive models, incorporating new information when it is available. Integrating new information may also benefit from the drift detection, and the occurrence of a drift may suggest eventual corrective actions to the model. Some specifics of the financial problems may turn the models quite stable along time, which is the case of the scorecard presented in this research. A static learning setting was at the basis of the model with the best discriminatory power. It also becomes apparent that some sort of time discretization may turn useless in some applications and may lead to nonsense or suboptimal forecasts. In this problem, as there is not an intra-annual seasonality of default, the practical meaning of a monthly prediction is may be debatable. Credit risk assessment is one area where the data mining and forecasting tools have largely expanded over the last years. However, there 82 are a few areas where the use of these tools should focus essentially on providing a direction, rather than providing a strict prediction. There are a number of possible directions that no model that looks just into the past can enlighten about the future. This includes the directions driven by the business strategy of the bank (e.g. expanding the branch network, to offer a new product, or to merge with another financial institution). The same applies to the occurrence of extreme or rare events, like those that were roused by the recent financial crisis. The new paradigm of forecasting turns out to be looking at hidden streams in the present signal and understand how they will possibly direct an event into the future. We propose a two-stage model for dealing with temporal degradation of credit scoring, which provided good results in a 1-year timeframe. However, it should also have a good performance in the long run. Therefore, future applications of this modelling framework should be tested in larger timeframes and consider lagged periods. Stress-testing this methodology should consider environments under major macroeconomic distress, or drifting populations resulting from expressive growth in the portfolios. 83 5. A new dynamic modelling framework for credit risk assessment Abstract - In this chapter we investigate the two mechanisms of memory, short-term (STM) and long-term memory (LTM), in the context of credit risk assessment. These components are fundamental to learning but are overlooked in credit risk modelling frameworks. As a consequence, current models are insensitive to changes, such as population drifts or periods of financial distress. We extend beyond the typical development of credit score modelling based in static learning settings to the use of dynamic learning frameworks. Exploring different amounts of memory enables a better adaptation of the model to the current states. This is particularly relevant during shocks, when limited memory is required for a rapid adjustment. At other times, a long memory is favoured. An empirical study relying on the Freddie Mac’s database, with 16.7 million mortgage loans granted in the U.S. from 1999 to 2013, suggests using a dynamic modelling of STM and LTM components to optimize current rating frameworks. Parts of this chapter are published in the Journal of Expert Systems with Applications in the paper titled “A new dynamic modeling framework for credit risk assessment”. ISSN: 09574174; DOI:10.1016/j.eswa.2015.09.055. A paper describing an application of the previous methodology to the Freddie Mac’s loans database was submitted to the Journal of Risk Model Validation with the title “Dynamic credit score modelling with short-term and long-term memories: the case of Freddie Mac’s database”, and is under review. Some sections are repeated through the dissertation to make the chapters self-contained. 84 5.1. Introduction More than half a century has passed since credit scoring models have been introduced to credit risk assessment and corporate bankruptcy prediction (Harold Bierman and Hausman, 1970, Altman, 1968, Smith, 1964, Myers and Forgy, 1963). With today’s advanced economies, a high proportion of the loan applications are automatically decided upon using frameworks where the credit score is the central, if not the unique, indicator of the borrowers’ credit risk. In the United States (U.S.), the FICO score is an industry standard, claimed to be used in 90% of lending decisions, to determine how much money each individual can borrow and to set the interest rate for each loan. In the OECD countries, banks that have adopted the Internal Ratings Based (IRB) approach, in Basel II Accord (Bank for International Settlements, 2006, Bank for International Settlements, 2004), are using their own credit scoring models as the basis of the regulatory capital calculation. A credit scoring model is meant to be an intelligent system. The output is a prediction about a given entity defaulting in a future period. In practice, one often uses a score that varies linearly in a positive range (e.g. FICO score varies in the range 300-850). In this arena, many frameworks, adaptations to real-life problems, and intertwining of base algorithms were, and continue to be, proposed in the literature, ranging from statistical approaches, to state-ofthe-art machine learning algorithms, from parametric models to nonparametric procedures, see the papers of Jones et al. (2015) and Orth (2013). Typical credit scoring systems are developed from static datasets. Subject to context specifics, and provided that certain requirements of the methods are met, a timeframe for the development is delimited at some point in the past. By referring to historical examples within such a timeframe, the model is 85 designed using a supervised learning approach. The resulting model is then used, possibly for several years, without further adaptation. As a consequence, traditional static credit scoring models are quite insensitive to changes within financial environments, like gradual or abrupt population changes caused by hidden transformations, or disturbances in periods of major financial distress. In line with this idea, Amato and Furnine (2004) found that ratings do not generally exhibit sensitivity to the business cycle. To some extent, credit scoring models development still need to better mimic the human learning established on experience. There are two basic mechanisms of memory, short-term memory (STM) and long-term memory (LTM), which are fundamental components of human experience and cognition. The former is easy to set up but readily forgotten; the latter may take longer to set up but tends to be more durable (Baddeley, 2012). The aim of this study is to find a clearer understanding of which type of memory configuration for the learning of credit scoring systems enables a rapid adaptation to changes. Hence, our analysis is set on two research questions: Is recent information relevant for improving forecasting accuracy? Does older information always improve forecasting accuracy? Consumers’ behaviour and default change over time in unpredictable ways. There are several types of evolution inside a population, for example population changes, that translate into changes in the distributions of the variables, affecting the models. The behaviour of the individuals and their ability to repay their debts change when the conditions within the economic cycle evolve. In good times, banks and borrowers tend to be overoptimistic about the future, whilst in times of recession banks are swamped with defaulted loans, high provisions, and tightened capital buffers. The former leads to more liberal credit policies and lower credit standards, the latter 92 to the new context. They are able to process examples batch-by-batch, or oneby-one, and update the prediction model after each batch, or after each example. Incremental models may rely on random previous examples, or on representative selected sets of examples, called incremental algorithms with partial memory (Maloof and Michalski, 2004). The challenge is to select an appropriate window size. 5.3. Case study Our research was conducted using the Freddie Mac’s single family mortgage loan-level database, first published in March 2013. It tracks the performance of 16.7 million of fully amortized 30-year fixed-rate mortgages loans in the U.S., granted between January 1st 1999 and March 31st 2013. Sharing this data follows the direction of the regulator, the Federal Housing Finance Agency (FHFA), as part of a larger effort to increase transparency and promote risk sharing. The primary goal of making this data available was to help investors build more accurate credit performance models in support of the risk sharing initiatives highlighted by the FHFA in the 2013 conservatorship scorecard. The dataset is live data updated over time, typically at the end of each quarter, with the application and performance data being summarized by month, from the application point until the most recent reporting period. 93 5.3.1. Origination data We considered a set of 16 variables that were available to the lenders at the time of the mortgage being granted, see Table 5.1. The release changes of the database are published online alongside a general user guide describing the full file layout and data dictionary (Freddie Mac, June 2013b). Freddie Mac’s information regarding the key loan attributes and performance metrics can be linked to our research in the aggregated summary statistics (Freddie Mac, June 2014b). Table 5.1: Case study U.S. - data available to the lenders at the loan application. Name Short description Type Credit score A number summarizing the borrower’s creditworthiness at the time of the origination date. Numeric First homebuyer flag Indicates whether the borrower is a first-time home buyer. Binary Metropolitan area Identified with the metropolitan statistical area (MSA) or metropolitan division (MD) based on census data. Treated as categorical Mortgage insurance percentage (MI%) The percentage of loss coverage that a mortgage insurer is providing to cover losses incurred as a result of a default on the loan, at the time of Freddie Mac’s purchase. For insured loans, the MI may vary between 1% and 55%. Numeric Number of units Denotes whether the mortgage is a one-, two-, three-, or four-unit property. Numeric Occupancy status Denotes whether the mortgage type is owner occupied, second home, or investment property. Categorical Original loan to value (LTV) Original mortgage loan amount divided by the lesser of the mortgaged property’s appraised value on the note date or its purchase price (in case of purchase or refinance mortgages). Numeric 94 Name Short description Type Ratios falling outside the range 6% and 105%, are disclosed as unknown. Original debt to income (DTI) ratio Debt to income ratio is based on the following calculation: Debt: the sum of the borrower's monthly debt payments, including monthly housing expenses that incorporate the mortgage payment the borrower is making, divided by; Income: the total monthly income used to underwrite as of the date of the origination of the mortgage loan. Ratios greater than 65% or unknown are passed as null values. Note: The disclosure of the dataset is subject to the widely varying standards originators use to verify borrowers’ assets and liabilities. Numeric Original amount The UPB of the mortgage on the note date, rounded to the nearest $1,000. Numeric Origination channel Indicates whether the channel at the origination of the mortgage is a retail lender, a broker or a correspondent. Situations where a third party origination is applicable but the seller did not specify the broker or correspondent are distinguished in the dataset. Categorical Prepayment penalty mortgage (PPM) Indicates whether the mortgage is a PPM. A PPM is a mortgage with respect to which the borrower is, or at any time has been, obligated to pay a penalty in the event of certain repayments of principal. Binary Property state A code identifying the state or territory within which the property securing the mortgage is located. Categorical Property type Denotes whether the property type secured by the mortgage is a condominium, leasehold, planned unit development (PUD), cooperative share, manufactured home, or Single Family home. Situations where the property state is unknown can be recognized in the dataset. Categorical Postal code The postal code for the location of the mortgaged property. Treated as categorical Loan purpose Indicates whether the mortgage loan is a purchase mortgage, a cash-out refinance mortgage, or a no cash-out refinance mortgage. Categorical Number of borrowers Identifies whether there is a single borrower or more who are obligated to repay the mortgage note secured by the mortgaged property. Treated as categorical 95 5.3.2. Performance data Loan performance information is provided on a monthly basis and includes the monthly loan balance, delinquency status and information regarding early termination events: voluntary prepayments in full; 180 days delinquency (“D180”); repurchases prior to D180; third-party sales prior to D180; short sales prior to D180; deeds-in-lieu of foreclosure prior to D180; real estate owned (REO) acquisition prior to D180. Specific credit performance information in the dataset includes voluntary prepayments and loans that were short sales, deeds-in-lieu of foreclosure, third party sales, and REOs. At the time of this research, data for performing loans and those that were up to 180 days delinquent was available through June 30th 2013. From the time it was granted until the most recent reporting period, there is a complete monthly historical report of the debt service for each loan, containing the following: Exposure at default value - ending balance as reported by the servicer for the corresponding monthly reporting period. Loan delinquency status - number of days that the borrower is delinquent, based on the due date of the last paid instalment reported by the servicers to Freddie Mac, calculated under the Mortgage Bankers Association (MBA) method. A code is used indicating the reason why the loan's balance was reduced to zero, in the following cases: - Prepaid or matured (voluntary payoff); - Foreclosed (short sale, third party sale, charge off or note sale); - Repurchased prior to property disposition, or; 96 - Real-estate owned (REO) disposition. We consider that a borrower defaulted if he was, at any point, 90 or more (90+) days delinquent, the typical definition used under Basel II. Later, in section 5.4, we will describe the construction of scorecards based on a supervised learning procedure and a binary target, where a borrower is assigned to the “bad” class, if he defaulted, and is assigned to the “good” class otherwise. 5.4. Modelling framework 5.4.1. Adaptive modelling The dynamic modelling framework implemented in this research considers that data is processed batch-by-batch, as illustrated in Fig. 5.1. Sequentially, every year, a new model is built from a previously selected window, including the most recent year. To have sufficient performance window length we chose not to use loans granted from 2012 onwards. In each model retraining - learning unit - we use a static setting. Each year, instances for modelling are selected from all previously available batches, according to a selection process. We use instance selection methods to test the hypothesis under investigation. Two methods were implemented – a LTM and a STM windowing configuration with a forgetting mechanism. 97 The LTM windowing configuration assumes that the learning algorithm generates the model based on all previous instances (Fig. 5.1(a)). The process is incremental, therefore every time a new instance arises, it is added to the training set, and a new model is built. This scheme should be appropriate to detect mild drifts, but it is unable to adapt rapidly to major changes. Models of this type should perform reasonably well in stable environments. A shortcoming of this incremental scheme is that the training dataset quickly expands, and this may require a huge storage capacity. In the STM windowing configuration, the model development uses the most recent window. With this scheme, Fig. 5.1(b), a new model is built in each new batch, by forgetting past examples. The fundamental assumption is that past examples have low correlation with the current default. Models of this type should quickly adapt to changes. A downfall of this method is that it often lacks the ability to generalize in stable conditions. (a) LTM. (b) STM. Fig. 5.1. Case study U.S. - adaptive learning windowing configurations. 98 5.4.2. Constructing the scorecards The classifier corresponding to each learning unit is a scorecard. Generalized Additive Models (GAM), introduced by Hastie and Tibshirani, are an extension of Generalized Linear Models (GLM) which, in turn, are an extension of Linear Regression (LR). Scorecards are GAMs, where the individual functions are piece-wise constant. The general approach to scorecard development involves the binning of the predictive variables and the optimization of the weight of each binned characteristic (Silva and Cardoso, 2015). A common practice is to compute the weights in two steps. Firstly, for each characteristic, the relative importance (score) of each bin is estimated; then, the relative importance of each characteristic is optimized. A standard way to estimate the relative importance of each bin is by using the weight of evidence (WoE) in the complete training dataset WoEi=ln(nGinG ⁄ nBinB ⁄), (3) where 𝑛𝐺𝑖 and 𝑛𝐵𝑖 are respectively the number of non-defaulted borrowers (good class) in the bin i and the number of defaulted borrowers (bad class) in the bin i, and 𝑛𝐺 and 𝑛𝐵 are respectively the total number of non-defaulted borrowers and total number of defaulted borrowers in the population sample. The larger the WoE is, the higher is the proportion of good borrowers in the bin. Numerical variables were firstly binned. Cases where the calculation of the WoE rendered impossible, i.e. no borrower following in one of the classes, are given an average value. The same rule is applied to values out of the expected ranges. The strength of each potential characteristic is measured using the information value (IV) in the training dataset 99 IV=∑( nGinG ⁄−nBinB ⁄)WoEi n i=1 , (4) where n is the number of bins in the characteristic. The higher the IV is, the higher is the relative importance of the characteristic in a univariate basis. Finally, the design of the scorecard is concluded by optimizing the weight of each characteristic using a linear model, as described in Silva and Cardoso (2015). The scorecard design is wrapped in a forward feature selection process to find the optimal subset of characteristics. The selection process stops when no other characteristic adds significant contribution to the information value (IV) of the model. In this application the stopping criterion was set for a minimum increment of 0.03 in the IV. Tables 5.2 and 5.3 show the marginal contribution of the characteristics in each model adjustment, respectively, in the LTM and in the STM memory configurations. Cells are highlighted in grey if the characteristic was selected in the model adjustment. It’s worth noticing that, in the LTM configuration, the optimal subset of characteristics is more stable, and that the adjusted models tend to select a smaller number of characteristics. The STM configuration often leads to an adaptation based on a larger set of characteristics. For the conclusions drawn from the experimental design to have validity, the same design process, as well as the same set of 16 potential predictors, was used in the learning units of both memory configurations (LTM and STM). In so doing, the difference in the performance of the models should be only due to the different time windows lengths. 100 Table 5.2: Case study U.S. - Marginal contribution in each LTM learning unit. Characteristic 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Credit score 1,238 1,325 1,393 1,586 1,586 1,527 1,502 1,459 1,307 1,220 1,288 1,317 1,335 First homebuyer flag 0,000 0,001 0,006 0,002 0,002 0,000 0,000 0,000 0,001 0,002 0,003 0,003 0,003 Metropolitan area 0,080 0,071 0,056 0,043 0,043 0,028 0,029 0,026 0,028 0,021 0,021 0,022 0,021 Mortgage insurance percentage 0,002 0,007 0,001 0,000 0,000 0,000 0,000 0,000 0,001 0,002 0,002 0,002 0,002 Number of units 0,006 0,000 0,001 0,000 0,000 0,000 0,001 0,001 0,001 0,003 0,004 0,003 0,003 Occupancy status 0,018 0,059 0,079 0,080 0,080 0,051 0,061 0,050 0,057 0,067 0,067 0,064 0,063 Original debt to income (DTI) ratio 0,051 0,016 0,020 0,014 0,014 0,022 0,021 0,024 0,039 0,069 0,077 0,084 0,088 Original amount 0,100 0,026 0,013 0,027 0,027 0,044 0,025 0,021 0,011 0,005 0,005 0,005 0,005 Original loan to value (LTV) 0,159 0,130 0,203 0,245 0,245 0,264 0,236 0,226 0,247 0,259 0,261 0,258 0,259 Origination channel 0,085 0,064 0,090 0,102 0,102 0,093 0,075 0,073 0,080 0,166 0,121 0,102 0,095 Prepayment penalty mortgage 0,005 0,003 0,006 0,007 0,007 0,007 0,006 0,005 0,003 0,013 0,019 0,021 0,024 Property state 0,082 0,107 0,080 0,086 0,086 0,056 0,167 0,137 0,090 0,075 0,074 0,074 0,074 Property type 0,025 0,000 0,000 0,000 0,000 0,000 0,029 0,000 0,024 0,000 0,018 0,019 0,019 Postal code 0,039 0,064 0,029 0,022 0,022 0,019 0,017 0,018 0,016 0,016 0,018 0,018 0,019 Loan purpose 0,021 0,017 0,017 0,000 0,000 0,013 0,011 0,012 0,022 0,014 0,015 0,015 0,017 Number of borrowers 0,335 0,492 0,446 0,462 0,462 0,391 0,407 0,421 0,422 0,442 0,449 0,448 0,452 Learning unit divergence 2,245 2,381 2,440 2,676 2,676 2,515 2,586 2,474 2,350 2,374 2,440 2,453 2,478 Characteristics in the model 9 8 7 7 7 7 6 6 7 7 7 7 7 Table 5.3: Case study U.S. - Marginal contribution in each STM learning unit. Characteristic 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Credit score 1,238 1,430 1,561 1,727 1,287 1,356 1,310 1,310 0,985 1,279 1,402 1,340 1,078 First homebuyer flag 0,000 0,007 0,008 0,008 0,001 0,000 0,001 0,001 0,000 0,000 0,000 0,000 0,026 Metropolitan area 0,080 0,022 0,046 0,046 0,072 0,070 0,088 0,088 0,029 0,025 0,207 0,254 0,365 Mortgage insurance percentage 0,002 0,258 0,289 0,289 0,003 0,000 0,003 0,003 0,001 0,004 0,000 0,024 0,000 Number of units 0,006 0,001 0,001 0,001 0,008 0,001 0,003 0,003 0,004 0,004 0,004 0,000 0,000 Occupancy status 0,018 0,062 0,113 0,113 0,000 0,004 0,011 0,011 0,049 0,090 0,017 0,003 0,002 Original debt to income (DTI) ratio 0,051 0,012 0,026 0,014 0,041 0,043 0,022 0,022 0,042 0,055 0,250 0,264 0,347 Original amount 0,100 0,013 0,047 0,025 0,056 0,064 0,032 0,032 0,003 0,158 0,011 0,193 0,194 Original loan to value (LTV) 0,159 0,008 0,013 0,013 0,304 0,247 0,119 0,119 0,404 0,322 0,224 0,003 0,318 Origination channel 0,085 0,085 0,174 0,174 0,072 0,040 0,015 0,015 0,076 0,058 0,043 0,020 0,015 Prepayment penalty mortgage 0,005 0,012 0,014 0,010 0,008 0,001 0,006 0,006 0,002 0,084 0,027 0,030 0,084 Property state 0,082 0,080 0,173 0,173 0,062 0,286 0,308 0,308 0,240 0,234 0,195 0,363 0,411 Property type 0,025 0,000 0,025 0,047 0,000 0,041 0,018 0,018 0,018 0,024 0,024 0,063 0,074 Postal code 0,039 0,022 0,014 0,026 0,068 0,053 0,027 0,027 0,026 0,020 0,065 0,162 0,160 Loan purpose 0,021 0,109 0,006 0,006 0,007 0,023 0,013 0,013 0,024 0,025 0,027 0,064 0,131 Number of borrowers 0,335 0,427 0,448 0,448 0,372 0,334 0,948 0,948 0,327 0,423 0,308 0,357 0,466 Learning unit divergence 2,2446 2,5486 2,9576 3,1196 2,3596 2,563 2,9224 2,9224 2,2312 2,8034 2,801 3,137 8 3,670 5 Characteristics in the model 9 7 8 8 9 10 6 6 7 9 8 10 11 The performance of the model is measured with the Gini coefficient, equivalent to the area under the ROC curve (AUC). It refers to the global quality of the credit scoring model, and may range between -1 and 1. The 101 perfect scoring model fully distinguishes the two target classes, good and bad, and has a Gini index equal to 1. A model with a random output has a Gini coefficient equal to 0. If the coefficient is negative, then the scores have a reverse meaning. An extreme case of -1 would mean that all examples of the good class are being predicted as bad, and vice-versa. In this case, the perfect model can be achieved just by switching the prediction. 5.5. Results We assessed the performance of the models sequentially learnt through the origination years 1999 to 2011. For each model rebuilding, the performance of the new model was measured in two sets: the modelling test set, containing a 20% random portion of the loans granted in the development year, and the set of loans granted in the following year, an out-of-sample performance. The vintage curves presented in a previous study of Landy, Ashworth and Yang (2014) suggest that the cumulative default rates of this portfolio reached a plateau by the fifth year. Since most of the default events occurred between the first and the fifth year after the loan had been granted, we assumed that the performance measures of the models should be calculated within this timeframe. Therefore, despite the fact that the models’ learning considered a fixed target concept - a borrower finding himself 90+ days delinquent at any point in a given timeframe after underwriting a loan - performance was measured in five annually-incremental performance windows, from a 1-year to a 5-year performance window after the loan had been granted. In so doing, our aim is to bring awareness to the true performance of the models over the most relevant part of the life of the asset, rather than just interpreting the 1- 108 loans that were at the epicentre of the global crisis, the Freddie Mac’s single family mortgage loan-level dataset, first published in 2013. We did not attempt to challenge the existing adaptive modelling techniques. Instead, we aimed at using a straightforward adaptive learning framework to explicitly exhibit the STM and LTM capabilities in model learning. Two plain assumptions are confirmed in our investigation: newest data consistently improves forecasting accuracy, and STM allows a quick adaptation to changes. Older information did not improve forecasting accuracy, but no general rule can be made, since it may be an outcome of the context specifics. Although we had assumed otherwise, our empirical study did not reveal that the LTM outperforms the STM during stable phases. We speculate that this may have been a consequence of having used an insufficiently short window in the STM configuration. Our research presents renewed relevant empirical evidence that traditional modelling frameworks significantly degrade over time and that the models’ predictive effectiveness is largely improved when adaptive learning frameworks are applied. There are some real business problems with rebuilding models over time. Firstly, lenders have little incentive to enhance the existing rating systems’ frameworks because it is expensive and time-consuming to build new scorecards. The scorecards need to be internally tested and validated, and then regulators need to approve them. Secondly, regulators still promote models whose coefficients do not change over time. This is one area where new evidence such as we have presented, might help. Our ideas for future work include trying to use ensembles of models that have been learnt from the past, instead of using the entire period to learn a new model. This has two major advantages. Firstly, a smaller sample is required for relearning the model, while still keeping memory from the past. Secondly, a model that depends on 109 the previous assessments is more palatable; hence, it is more likely to be accepted. Another viable option is to develop a straightforward mechanism for modelling the link between the two components of memory identified in this study – LTM and STM. Regarding the STM, a prior selection of the window length seems appropriate and should be employed to optimize adaptation ability. 110 111 6. Risk management with dynamic defaults and under the Basel capital rules Abstract - Stress-testing is an important risk management tool used by banks and supervisors. The recent deployment of large databases enables the realistic evaluation of financial models. We present a first application of Freddie Mac’s database recently released in 2013 to stress-testing, which can easily be replicated to other real-loans portfolios. Our proposal includes a first implementation of a return on risk-adjusted equity model embedded in the contemporary capital regulatory rules. Under this setting we analyse the impact of the Probability of Default (PD) and the Loss Given Default (LGD) in return on lending under the most extreme adverse circumstances. Subprime lending was mostly banned from the primary markets. We claim that this is an overreaction to the global financial crisis. We find that much of the disturbances in the return on lending only occur for high LGD values. For LGD below 25%, simulations show that lending to borrowers with lower scores produces positive returns in the long-run. If sufficiently mature, these loans can boost portfolios' compositions, because they are less exposed to early repayments. So, rather than strictly declining these loans, regulators and lenders should ascertain the LGD boundaries under which the bank operates, to drive lending policies. Parts of this chapter are published in the paper titled “Stress-testing the return on lending under real extreme adverse circumstances” which was presented at the EFMA Annual Meetings 2015. Taking in consideration the comments received, the paper was submitted with minor changes to the Journal of Credit Risk, with the title “Stress-testing the return on lending in the aftermath of the Crisis under the Basel Capital Rules”, where it is under review. Some sections are repeated through the dissertation to make the chapters self-contained. 112 6.1. Introduction The subprime mortgage lending crisis in the U.S. came to public’s attention when home foreclosures begun to rise in 2006 and moved out of control in 2007. A large decline in home prices prompted a devaluation housing-related securities and an unprecedented rise in mortgage delinquencies. Worldwide, banks' liquidity has plummeted with a significant disruption of the financing of businesses and consumers. This brought dramatic changes to financial regulation and banking supervision, including in the area of mortgages and consumer credit. Simultaneously, consumer financial protection regulation has been strengthened and expanded, and consumer financial behaviour has been changing so far. Our study develops an empirical risk measure that relies heavily on the historical experience of delinquencies in this evolving landscape. In so doing, we contribute to enhanced knowledge in the area of credit risk assessment, which is relevant for setting regulatory and lending policies. We are focused on the study of the disturbances that affect the return on lending in real-world environments. The research question is if lending to the borrowers with the lower scores at the time of the loan application always produces negative returns in the long-run. 6.1.1. Stress-testing the return on lending under the Basel capital rules Since 21 March 2013, Freddie Mac is making available loan-level credit performance data on a portion of fully amortized 30-year fixed-rate mortgages 113 that the company purchased or guaranteed since 1999. The data is provided in a “living” dataset and by June 2014, the database covered over 16.7 million of fully amortized, 30-year fixed-rate mortgages in the U.S., originated between 1999 and the first quarter of 2013. Based on the historical observed delinquencies, we employ stress-testing as an attempt to project the returns under realistic extreme adverse economic scenarios. Returns are modelled under the current Basel capital rules, and so, the impact analysis is made considering two settings – applying the risk-weights as defined by the Basel Committee, and considering the parameters of the Final Rule under the Basel III for the U.S. banking. We stand on some basic ideas of Saunders and Allen (2010) and on the model of Ruthenberg and Landskroner (2008), adapting the formulation of the model to our problem. We formulate the Basel Capital Requirements based on the work of Antão and Lacerda (2011), and in the Basel Committee regulation under Basel II and Basel III Accords (Bank for International Settlements, 2006, Bank for International Settlements, 2010). 6.1.2. Credit scoring – a central input in today’s lending 6.1.2.1. Automated decision-making Under the Basel capital rules, banks tailor their strategies to suitably remunerate their shareholders. In tailoring their activity strategies, commercial banks have two important decisions: selecting the borrowers’ 114 desired risk profile, and positioning the pricing strategy, subjected to the market competitors and to regulatory boundaries. When selecting the borrower’s desired risk profile, lenders determine if the risk of lending to a borrower is acceptable under certain parameters of credit risk, borrower’s credit capacity and collateral evaluation. In retail lending, a great proportion of the loan applications are automatically evaluated. In this setting, credit score is the central, if not unique, indicator of the borrowers’ credit risk, either when the credit decision assessment is fully automatic or when it is an input for human decision. A person without a credit score or with a low score (meaning high risk) is unlikely to have credit, whilst an application of a person with a high score has good chances to be accepted. An analysis on the causes and effects of the mortgage meltdown states that in 2007, 40% of all subprime loans have been generated by automatic underwritings in the U.S. This has been associated to lax controls in the underwriting processes (Bianco, 2008). Automated processes meant fasters decision, but less documentation scrutiny. 6.1.2.2. Risk-based pricing Banks increasingly use risk-based pricing models to price loans when positioning their pricing strategies, which are also moving towards credit score’s over-dependence. In the U.S., since its introduction 20 years ago, FICO score is calculated from the information available in the individuals’ credit bureau reports, and has become an industry standard. It is claimed to be used in 90% of lending decisions, to determine how much money each 115 individual can borrow, and how much interest he will pay. As a result of the industry standards, the performance of credit loans depends on the credit scoring models accuracy, both in the short-term as in the long-run predictions. In 2007 and 2008, the delinquency rate in the mortgage loans in the U.S. rose sharply, both in borrowers in the lower scores as in the highest scores bands, showing that the actual risk of these borrowers has been underestimated. 6.1.3. The Freddie Mac’s database – preliminary findings Anderson, Scott, and Janet Jozwik (2014) proposed a framework for developing a credit model based on the Freddie Mac’s dataset. For a 180-days delinquent target event, the authors conclude that much of the variation in credit performance across loans and over different stages of the economic cycle is explained by loan-level variables. Unsurprisingly, by adding factors to capture broader macroeconomic effects and the quality of underwriting, they significantly improve the model. Goodman, Landy, Ashworth and Yang (2014) present an exploratory paper providing a first look through the data, to find potential implications for guarantee pricing. The authors show the vintage composition as a percentage of the initial balance in a cross-analysis of the original borrowers’ FICO score by the original loan to value (LTV). They follow the cumulative default in three groups in the score ranges 300 to 700, 700 to 750 and 750 to 850 crossed by the original LTV in selected buckets. The authors conclude that default rates are dramatically higher on higher LTV/lower scores, and so, they suggest that investors should look not only at the average LTV and FICO scores, but also at the FICO/LTV loans’ distribution. The authors conjecture that pricing these portfolios by looking at 116 averages should have led to under-priced default risk, but they do not present evidence. Sousa, Gama and Brandão (2015a) investigate the same database and find evidence that the first year cumulative default of the borrowers with score 650 or higher has almost tripled in first years of the last global crisis, suggesting that credit risk may has been under-priced in these cases. They show that default rates increased sharply during the crisis, but it did not increase uniformly along the range of borrowers’ credit scores. They also show that, two years after the crash, lending decision threshold moved markedly to borrowers with scores higher than 625, which led to an increase in concentration of lending in the individuals in the highest score bands. Although this is a reasonable prudential measure, excessive lending bias and concentration towards the highest scores require more precise default estimation to correctly price credit risk. 6.1.4. Contributions of this study Discussion is being pushed towards risk-based pricing designs under the current capital rules. Previous studies (Anderson and Jozwik, 2014, Goodman et al., 2014, Sousa et al., 2015a) suggest that risk-based pricing models used within today’s lending are highly dependent on credit scores, which in turn are increasingly ruled under the Basel capital regulation. Behn, Haselmann and Wachtel (2015) use a quasi-experimental research design to examine the effect of model-based capital regulation on the pro-cyclicality of bank lending and firms' access to funds. Complementing the previous works, we focus on the performance of returns in different arrangements of borrowers’ credit risks, through contrasting 117 phases of the economic and business cycles, including the years of exacerbated financial distress of the last global crisis. This chapter follows in section 6.2 with a description of the return on riskadjusted equity model that is used in our study. We describe the regulatory environment, under the current capital regulation, and the base concepts that rule the model. In section 6.3, we describe the conditions of our empirical study. Firstly, we make an overview of the database used in our experimental study. Secondly, we provide a descriptive analysis regarding the interest rates setting, and about the lending evolution over the analysed period. Finally, we describe the main assumptions of the stress-testing exercise of this research. Section 6.4 provides the results for nine tested scenarios. We depict a pessimistic scenario that has been drawn from the real historical default rates of mortgage loans in the U.S. during the global crisis. Conclusions are presented in section 6.5. 6.2. Return on risk-adjusted equity model Our model assumes a commercial bank that operates in the primary market, raising deposits from the clients and extending credit to the public. It also operates in the secondary market, where it transacts with other commercial banks, with the central bank, and in the financial markets. The bank holds regulatory capital as defined under the internal ratings-based (IRB) issued in Basel II as a cushion against unexpected losses. We assume that the bank uses a risk-based pricing (RBP) model to price loans. The objective function is to maximize the expected profit as a function of credit risk, based on the decision 124 capital-related provisions of the Dodd-Frank Act. In the E.U., Basel Accords have been introduced via the Capital Requirements Directive (CRD). This Directive provides a common framework for implementation, but allows for national discretions. One example of a national discretion is the LTV limit for qualifying residential mortgages with the preferential risk weighting of 35%. In the UK, for example, this limit has been set at 80% by the Financial Services Authority (FSA). For a bank using the IRB approach without any changes rw is 12.5×1.06×K. Equation (9) can be rewritten as: 𝐴=𝑠𝑟×𝑟𝑤𝐴+(1−𝑠𝑟×𝑟𝑤)𝐴 (11) The cost of equity, 𝐶𝐸, can be evaluated according to the return required by the shareholders, for which we assume the profitability measure return on equity, 𝑟𝐸. 𝐶𝐸=𝑟𝐸𝐸. (12) Equity is the costliest way of financing. So, financial institutions using the IRB benefit from lower costs of funding when lending to entities of better risks, i.e. lower PD and lower LGD. In the other extreme their cost of funding is higher when lending to entities of worst risks, i.e. higher PD and higher LGD. This can easily be recognised from the monotonic behaviour of the 𝑟𝑤 factor when varying these two parameters, which we illustrate with an example in Fig 6.1. 125 Fig. 6.1: Illustrative IRB risk-weights for residential retail. Source: Bank for International Settlements (2006). On the other side, the cost of debt, 𝐶𝐷, is 𝐶𝐷=𝑟𝐷𝐷, (13) where 𝑟𝐷 is the price of debt, considering the structure of financing and price that the bank can get in wholesale funding (central banks and markets) and from clients’ deposits. 126 6.2.1.6. Expected loss Our model assumes that when a borrower enters in default then from that point onwards he will not leave the default status. If the 𝑃𝐷𝑖, i=1.. N, is the probability of the borrower entering in default in the month 𝑖 and the LGD is the loss given default conditioned on the default event, then the expected loss for that borrower in that month, 𝐸𝐿𝑖, is 𝐸𝐿𝑖=𝑃𝐷𝑖×𝐿𝐺𝐷×𝐸𝐴𝐷𝑖 (14) The cumulative default until the end of the loan is equal to the sum of the probabilities of default in each year, as illustrated in Fig. 6.2. Fig. 6.2: Illustrative example of a cumulative default. In the remainder of the chapter, we will assume that both the PD and the LGD parameters are available for each borrower at the time of the underwriting. 0% 10% 20% 30% 40% 1 2 3 4 5 Cumulative default (%) Years after underwritting 𝑃𝐷1 𝑃𝐷2 𝑃𝐷3 𝑃𝐷4 𝑃𝐷 5 127 6.2.2. The model The starting point for achieving the return expected for the loan is the equality: Profits = Costs (15) For the costs associated with the loan, we consider the cost of funding and the losses associated with credit default events, the expected loss (EL). Other costs, such as administrative or infrastructure costs, are not considered in this model, despite of being relevant for the banks’ activity based costing. Costs=𝐶𝐸+𝐶𝐷+𝐸𝐿 (16) Profits include the fees and the interests charged on the loan. We assume that the interest rate consists of a market reference rate, rr, plus a spread, 𝑠𝑝. The spread component is intended to cover a profit margin and the risks associated with the loan, such as credit risk, liquidity risk, market risk, operational risk, and other risks as may be identified by the bank from time to time. Traditionally, fees, 𝑓, can be either ad valorem or bullet. The former is equivalent to using a spread. The latter can be converted into a spread prior to the calculation of profits, based on the maturity of the loan and the frequency of payments. Profits=(𝑟𝑟+𝑠𝑝)𝐴+𝑓 (17) Considering the equalities in (16) and (17), equation (15) is rewritten as: (𝑟𝑟+𝑠𝑝)𝐴+𝑓 = 𝐶𝐸+𝐶𝐷+𝐸𝐿 (18) 128 equivalent to: (𝑟𝑟+𝑠𝑝)𝐴+ 𝑓 = 𝑟𝐸𝐸+𝑟𝐷𝐷+𝐸𝐿 (19) Isolating the components of spread intended to cover the credit and liquidity risks, applied to the total amount to be financed, the previous equation is rewritten as: (𝑟𝑟+𝑠𝑝𝑐+𝑠𝑝𝑙+𝑠𝑝𝑜)𝐴 + 𝑓= 𝑟𝐸𝐸+𝑟𝐷𝐷+𝐸𝐿 (20) where, 𝑠𝑝𝑐 is the credit risk spread, 𝑠𝑝𝑙 is the liquidity spread, and 𝑠𝑝𝑜 is the spread that covers the profit margin and remaining risks, like operational or country risk. 6.2.2.1. Credit risk spread Equation (20) can be rewritten as: 𝑠𝑝𝑐𝐴= 𝑟𝐸𝐸+𝑟𝐷(𝐴−𝐸)+𝐸𝐿−(𝑟𝑟+𝑠𝑝𝑙+𝑠𝑝𝑜)𝐴−𝑓 (21) Rearranging the arguments and isolating the credit risk spread, this is equivalent to: 𝑠𝑝𝑐= 𝐸 𝐴(𝑟𝐸−𝑟𝐷)+(𝑟𝐷−𝑟𝑟−𝑠𝑝𝑙−𝑠𝑝𝑜)+𝐸𝐿−𝑓 𝐴 (22) 129 Considering the equality in (10), then the credit spread that allows an adequate return on equity 6 is: 𝑠𝑝𝑐= 𝑠𝑟×𝑟𝑤(𝑟𝐸−𝑟𝐷)+(𝑟𝐷−𝑟𝑟−𝑠𝑝𝑙−𝑠𝑝𝑜)+𝐸𝐿−𝑓 𝐴 (23) If the bank funds the debt by the market reference rate, i.e. 𝑟𝐷=𝑟𝑟, then the credit risk spread is: 𝑠𝑝𝑐= 𝑠𝑟.𝑟𝑤.(𝑟𝐸−𝑟𝑟)−(𝑠𝑝𝑙+𝑠𝑝𝑜)+𝐸𝐿−𝑓 𝐴 (24) 6.2.2.2. Return on equity of a loan Rearranging the arguments in equation (20), and isolating 𝑟𝐸, then the return on equity rate needed to fund the loan is: 𝑟𝐸=1 𝐸(𝑠𝑝𝑐𝐴−𝐸𝐿)− 𝐴 𝐸(𝑟𝐷−𝑟𝑟−𝑠𝑝𝑙−𝑠𝑝𝑜)+𝑟𝐷+𝑓 𝐸 (25) Or, stated equivalently: 𝑟𝐸=1 𝑠𝑟.𝑟𝑤[(𝑠𝑝𝑐−𝑟𝐷+𝑟𝑟+𝑠𝑝𝑙+𝑠𝑝𝑜)−𝐸𝐿−𝑓 𝐴]+𝑟𝐷 (26) Likewise, if the bank funds the debt by the market reference rate, i.e. 𝑟𝐷=𝑟𝑟, then the return needed to fund a loan is 𝑟𝐸=1 𝑠𝑟.𝑟𝑤[(𝑠𝑝𝑐+𝑠𝑝𝑙+𝑠𝑝𝑜)−𝐸𝐿−𝑓 𝐴]+𝑟𝑟. (27) 6 Here we consider the return before taxes. 130 6.3. Empirical study 6.3.1. Data The empirical study summarized in this chapter is based on the Freddie Mac’s single family mortgage loan-level dataset, first published in March 2013. We have followed the performance of 16.7 million of fully amortized 30-year fixed-rate mortgages loans in the U.S., granted between January 1st 1999 and March 31st 2013. Loans performance has been measured in a monthly basis and, at the time of this research, data for performing loans and those that were up to 180 days delinquent were available through September 30th 2013. The dataset is updated over time, typically at the end of each quarter. Release changes, as well as a general user guide describing the file layout and data dictionary, are recorded online (Freddie Mac, June 2014a). Freddie Mac’s information regarding the key loan attributes and performance metrics can be linked to this research in the aggregated summary statistics (Freddie Mac, June 2014b). Data of the original datasets were aggregated by the origination year. Scores may vary in the range 300-850, or be unknown. Situations where the score is unknown are described by Freddie Mac’s Corporation (June 2013b), who refers that these may be an outcome of the sellers’ reduced level of verification. We divided the range of possible scores into equidistant intervals of 25, except for the lower and upper bounds. To have dimension, these bounds were aggregated in the buckets [300, 550[ and [800, 850[, respectively. 131 6.3.2. Interest rates setting Mortgage interest rates are affected by many factors. In the United States they are heavily influenced by the monetary policies of the Federal Reserve Board’s Federal Open Market Committee (FOMC). Trends in interest rates on longer financial instruments, such as mortgages, typically follow the fluctuation of the 10-year Treasury note yield. Fig. 6.3 shows the evolution of the 10-year Treasury constant maturity rate over the period 2005 through 2012, and the 10-year constant maturity advance rates authorized by the Federal Housing Finance Agency (FHFA) for the advance pricing. Fig. 6.3: Case study U.S. - Rates evolution in the period 1999-2012. Sources: Freddie Mac’s single-family loan level dataset, FHFA historical advance rates and FED 10-year Treasury note yield. These rates serve as a reference for the 30-year fixed-rate mortgage loans. Their evolution is represented in Fig. 6.3, together with the average rate, the prime rate and the subprime rate. Borrowers’ scores are used to differentiate the interest rates of the mortgages, with the subprime rates being superior to the prime rates. In the period under analysis, the average rates are shifted to the prime rates (in Fig. 6.3. these lines are overlapped), because lending tends 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 19992000200120022003200420052006200720082009201020112012 Rate (%) Origination year Price at a low score Weighted average price Price at the highest score FHFA rate 10-Year Treasury Constant Maturity Rate 132 to be more driven to borrowers with the highest scores. The gap between the subprime and the prime rates slightly increased in the aftermath of the crisis, between 2008 and 2011, as illustrated in Fig. 6.4. Fig. 6.4: Case study U.S. - Gap subprime - prime rate, 1999-2012. Source: Freddie Mac single-family loan level dataset. Between 2004 and 2007, spreads placed around 170 bps for the prime loans and in the average portfolio, and around 190 bps for the subprime loans (Fig. 6.5). Spreads increased in the aftermath of the crisis, and reached a peak in 2008 (Fig. 6.5). A recent report of the Federal Housing Finance Agency says that a 170 bps spread is expected from 2015 through 2017 (Schultz, August 2014) to cover liquidity and credit risks. Fig. 6.5: Case study U.S. - 30-year mortgage spreads in the period 1999-2012. 0.00 0.10 0.20 0.30 0.40 0.50 0.60 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 SUBPRIME RATE - PRIME RATE Origination year 130 180 230 280 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 Spread (bps) Subprime rate Weighted average rate Prime rate 133 6.3.3. Stress testing scenarios 6.3.3.1. Expect margin and expected losses assumptions We have analysed the performance of the returns in each year after the loan has been underwritten. Losses have been empirically assessed based on the past observed default rates in each score bucket. Rather than just using the average default in the period, as traditional approaches often do, we have focused on the two possible extreme circumstances – an optimistic scenario and a pessimistic scenario. The optimistic scenario is ruled by the cumulative default rates by vintage of an origination year with a lower 1-year default rate (1999). The pessimistic scenario is ruled by the cumulative default rates by vintage of the origination year with the highest 1-year default rate (2008). We have also analysed an average scenario, as a reference, which is ruled on the weighted average annual cumulative default rates in the period 1999-2008. Default rates measured after 2008 were not considered in this analysis because the vintage curves could not be measured from the fifth year onwards (e.g. for the loans underwritten in 2009, only a 4-years vintage can be measured until 2013, and for the loans underwritten in 2012, only a 1-year vintage can be measured until 2013). For the purpose of this study, a borrower is considered to have entered in default after completing 90 consecutive days in delinquency. The vintage curves presented in a previous study of Landy, Ashworth and Yang (2014) suggest that the default rate by vintage have a plateau from the fifth year onwards. Hence, for this analysis we assume the measures of default rates in the first five years of the loan will reveal most of the expected default of the portfolio. The real default rates were measured