Portfolio optimization efficiency test considering data snooping bias

73 Business Systems Research | Vol. 11 No. 2 |2020 Portfolio Optimization Efficiency Test Considering Data Snooping Bias Aleš Kresta VSB – Technical University of Ostrava, Faculty of Economics, Czech Republic Anlan Wang VSB – Technical University of Ostrava, Faculty of Economics, Czech Republic Abstract Background: In the portfolio optimization area, most of the research is focused on insample portfolio optimization. One may ask a rational question of what the efficiency of the portfolio optimization strategy is and how to measure it. Objectives: The objective of the paper is to propose the approach to measuring the efficiency of the portfolio strategy based on the hypothesis inference methodology and considering a possible data snooping bias. The proposed approach is demonstrated on the Markowitz minimum variance model and the fuzzy probabilities minimum variance model. Methods/Approach: The proposed approach is based on a statistical test. The null hypothesis is that the analysed portfolio optimization strategy creates a portfolio randomly, while the alternative hypothesis is that an optimized portfolio is created in such a way that the risk of the portfolio is lowered. Results: It is found out that the analysed strategies indeed lower the risk of the portfolio during the market’s decline in the global financial crisis and in 94% of the time in the 2009-2019 period. Conclusions: The analysed strategies lower the risk of the portfolio in the out-of-sample period. Keywords: data snooping bias, financial crisis, hypothesis test, minimum-risk portfolio, portfolio optimization JEL main category: G11 JEL classification: G11, G17 Paper type: Research article Received: Jan 31, 2020 Accepted: Jul 06, 2020 Acknowledgments: The authors were supported through the Czech Science Foundation (GACR) under the project no. 18-13951S and the SGS research project of VSB-TU Ostrava under the project no. SP2020/11. The support is greatly appreciated. Citation: Kresta, A., Wang, A. (2020), “Portfolio Optimization Efficiency Test Considering Data Snooping Bias”, Business Systems Research, Vol. 11 No. 2, pp. 73-85. DOI: 10.2478/bsrj-2020-0016 74 Business Systems Research | Vol. 11 No. 2 |2020 Introduction Since the pioneering work of Markowitz (1952) that put out the foundations of modern portfolio theory, it is still a lively and open area with a lot of attention from both academics and practitioners. While academics generally focus more on the question of different models of portfolio selection, paying attention mostly to the theories and assumptions, practitioners focus also on the verification of particular strategies. The question they ask is what the efficiency of the portfolio optimization strategy is. The standard procedure is that the parameters of the model are estimated from the time series of returns in a historical period, which is usually called the in-sample period. The portfolio performance is then observed and measured in the period following the in-sample period. This observed period is usually called the out-of-sample period and the portfolio performance is generally lower in the out-of-sample period than in the in-sample period. In order to confirm the suitability of the proposed strategy, the out-of-sample performance is usually compared to the chosen benchmark. If the proposed strategy delivers better out-of-sample results than the benchmark, it is considered suitable. Although the application of the benchmark is commonly accepted, its informative value is rather poor – it answers the question of whether and how much the tested strategy overperformed the benchmark in one particular period. However, it does not say whether the overperformance is high enough to be considered as significant or whether it is just due to the randomness in data and luck. Also, another drawback of this simple approach is that it does not address the data snooping bias – if more strategies are analyzed, then it is more probable that the best strategy overperforming the benchmark is found. In this paper, we address this gap by proposing a rather different approach to verify the efficiency of the portfolio strategies. The approach is based on the hypothesis inference methodology. Put it simply, the principle is to generate many random portfolios and compare the out-of-sample performance of the analyzed strategy with performances of these random portfolios. Data snooping bias is also considered in the proposed approach. Moreover, in the empirical part of the paper, we investigate whether the classical minimum variance model (Markowitz, 1952) and fuzzy probabilities minimum variance model (Tanaka et al., 2000) decrease the risk in the out-of-sample period. The paper is divided as follows. The following section provides a brief literature review of applied benchmarks. Then, in the following section, the analyzed portfolio strategies are introduced and the method of testing the efficiency of portfolio strategies is presented. Empirical results are provided in the following section. First, the focus is given to the global financial crisis period. Then, the results on the rolling window basis in the period 2006-2019 are presented. The last sections provide a brief discussion of the results and conclusion to the paper. Literature review Various benchmarks can be found in scientific literature; however, there are few groups of benchmarks, which are generally applied. The first benchmark is 1/N strategy, which is also called a naive diversification strategy. The 1/N strategy is easy to implement because it is not necessary to estimate the future returns of the assets, the assets of the naive portfolio are invested with equal weights. DeMiguel et al. (2009) analyzed the differences in the performance of several optimization methods with that of the 1/N strategy, and they found that the effect of estimation error on return 75 Business Systems Research | Vol. 11 No. 2 |2020 probability distribution is large in those optimization models and this type of error can be avoided by using the 1/N weights. The second benchmark is the classical Mean-Variance model or Mean-VaR model. Since more real-life conditions have been considered, for example, the investors’ subjective preference, the transaction costs, the liquidity of the portfolio and so on, the classical Mean-Variance model is improved by incorporating various additional constraints, so the classical model becomes a widely used benchmark to compare with those enhanced approaches, see e.g. Fulga (2016), Ranković et al. (2016), Lwin et al. (2017), or Babazadeh and Esfahanipour (2019). It is also common to apply market indices as a benchmark to compare the performance of proposed methods. Solares et al. (2019) pointed out the main problem of applying market indices as the benchmark: the performance of portfolios is often compared to that of popular indices, but it is hard to reach the performance of the indices because there is a difference between characteristics of the stocks in the portfolios and the stocks contained in the index. So, to avoid this trap, it is recommended to construct the portfolio with only the stocks, which are selected from the components of the benchmark index. Besides considering one particular portfolio optimization approach as a benchmark, it is also necessary to apply a benchmark dataset when developing a new approach in the portfolio optimization problem. Most of the studies are based on either case studies or publicly available benchmarking datasets, see Kalayci et al. (2019). In the empirical analysis of this paper, the dataset of Dow Jones Industrial Average (DJIA) index is applied; it is a small dataset of 30 stocks while it is persuasive enough to explain the investment environment in the optimization problem. What’s more, the advantage of the small dataset is also its relatively smaller computational complexity. In the future, the small dataset of DJIA could be used as a benchmark dataset to compare with large-scale datasets, which can verify the applicability of the proposed models in complex environments and global markets. Methodology Applied Portfolio Models Portfolio models applied in the paper follow the classical mean-variance framework, i.e. only expected return (mean), its variance, and their inter-relationship are considered. Let us denote 𝑥 as the weight of 𝑖th asset in the portfolio. Short sales are excluded from the models, so the values of 𝑥 satisfies 0≤𝑥≤1 for all assets. If the expected return of 𝑖th asset is denoted as 𝐸(𝑅), then the expected return of a portfolio 𝐸(𝑅) is the weighted average of 𝐸(𝑅): 𝐸𝑅=∑𝑥∙   𝐸(𝑅)=𝑥∙𝐸(𝑅), (1) where 𝑁 is the total number of assets in the portfolio, 𝑥=[𝑥,𝑥,…,𝑥] is portfolio composition and 𝐸(𝑅)=[ 𝐸(𝑅),𝐸(𝑅),…,𝐸(𝑅)] is the vector of expected returns. The variance and standard deviation of the portfolio return are calculated by means of covariances 𝜎, of the asset returns for all asset pairs (𝑖,𝑗), 𝜎=∑ ∑ 𝑥∙𝜎, ∙𝑥     =𝑥∙𝐐∙𝑥, (2) 𝜎=𝜎, (3) where 𝐐 denotes covariance matrix, 𝐐=[𝜎,,𝑖=1,2,…,𝑁,𝑗=1,2,…,𝑁]. The minimumvariance portfolio can be found by solving the following quadratic optimization problem, minimize σ (4) 76 Business Systems Research | Vol. 11 No. 2 |2020 subject to ∑𝑥   =1 (5) 𝑥≥0,𝑖=1,…,𝑁 (6) The question is how to estimate the parameters of the return probability distributions. In the simplest approach, let us call it historical estimation, the characteristics of the observed sample distribution are calculated. Specifically, the expected returns are estimated as means of m observed historical returns, 𝐸(𝑟)=𝑟 =  ∑𝑟,   . (7) The covariance matrix is estimated in the same way, 𝑸=𝑸 =𝜎𝒊,𝒋 (8) 𝜎𝒊,𝒋 = ∑(𝑟,   −𝑟 )(𝑟, −𝑟 ) (9) The problem of historical estimation is the excessive sensitivity of the portfolio composition to errors in parameter estimates (due to the length of time series, the number of assets, etc.). The illustrative example can be found in DeMiguel et al. (2009). In the Bayesian approach, the estimation of the parameters of return probability distributions, i.e. the vector of expected returns 𝐸(𝑅) and the covariance matrix Q, considers the subjective (a priori) assumption of the shape of this distribution. The resulting (a posteriori) probability distribution is then a combination of the a priori assumption and the probability distribution of the observed sample. Although there are different possibilities for applying this approach, the methodology commonly referred to as the Bayes-Stein portfolio (BS) is applied in the paper. The foundations of this approach have already been laid by Stein (1956) and James and Stein (1961). The estimation suggested in Jordon (1986) is applied: 𝐸𝑟=(1−𝜉)∙ 𝑟 +𝜉∙𝑟, (10) ξ=  ()∙( )∙𝑸 ∙( ), (11) 𝑸 =𝑸 󰇡1+  󰇢+  ∙() 𝟏∙𝟏 𝟏×𝑸 ×𝟏, (12) 𝜁=∙ , (13) where 𝑟  is the historical estimate of the expected return of the ith asset, see (7), 𝑟 is the a priori expected return on the assets, 𝑁 is the number of assets and 𝑚 is the number of historical observations of returns (sample size), 𝑸  is the historically estimated covariance matrix, see (8). Jorion (1986) proposes to determine 𝑟 as the return of the minimum-variance portfolio. However, we rather consider the average expected return, which saves one optimization in the procedure. As can be seen, the shrinkage factor ξ depends on the number of assets 𝑁 (with an increasing number of assets the estimation error increases), the number of historical observations 𝑚 (the shorter the history, the higher the estimation error) and on the dispersion of estimated expected returns from a priori assumption (the greater the dispersion, the greater the estimation error). Another approach of how to handle uncertainty in probability distribution parameters is to apply the fuzzy theory. Tanaka et al. (2000) proposed two types of portfolio optimization models. The first model is based on fuzzy probabilities and aims to minimize the variance of the portfolio return while the latter utilizes possibility distributions and minimize the spread of the portfolio return. The first model is applied in the paper. The model considers not only historical returns {𝑟,𝑖=1,...,𝑁}, but also possibility grades {ℎ,𝑖=1,...,𝑁}, which represent a similarity between the future state of the stock market and the state of 𝑖th sample offered by experts. Given the historical returns and possibility grades, the fuzzy weighted expected return can be calculated as follows, 77 Business Systems Research | Vol. 11 No. 2 |2020 𝐸𝑟=∑∙,   ∑   , (14) and fuzzy weighted covariance matrix 𝐐𝑭=[𝜎, ,𝑖=1,…,𝑁,𝑗=1,…,𝑁] can be defined as: 𝜎, =∑󰇡,󰇢∙,󰇡󰇢∙   ∑   . (15) These estimates can be directly applied in the portfolio optimization model (4)-(6). Portfolio efficiency test The rational question one can ask is what the efficiency of the portfolio optimization strategy is. Let’s consider the portfolio strategy with observed out-of-sample performance 𝑝. The observed out-of-sample performance is, in fact, a realization of random variable 𝑃. Let us consider the following null hypothesis: the performance of the portfolio strategy is the same as of random strategy. The alternative hypothesis is that the performance of the portfolio strategy is better than of random strategy. The distribution of random variable 𝑃 under the null hypothesis can be obtained by generating random portfolio compositions and calculating their performances in the out-of-sample period. Then, the classical statistical inference approach is to calculate one-sided p-value: 𝑝=Pr(𝑃>𝑝), (16) in case that the performance measure should be maximized (Sharpe ratio, etc.) and 𝑝=Pr(𝑃<𝑝), (17) in case that the performance measure should be minimized (maximum drawdown, variance, etc.). If 𝑝 is smaller than the chosen significance level (0.05 or 0.01), the conclusion can be made that analyzed strategy performs better than random in the out-of-sample, i.e. the null hypothesis should be rejected and the alternative hypothesis should be accepted. The one-sided version of the test is applied because the aim is to prove that the performance of the analyzed strategy is better than random. Alternatively, a twosided version of the test can be applied, i.e. alternative hypothesis would take the form that the performance of the portfolio strategy is not the same as of random portfolio. However, in this case, rejecting the null hypothesis would only mean that the performance of the portfolio strategy is non-random, not knowing whether it is better or worse than random. However, there is one more issue that must be taken into account – data snooping bias also named as data mining bias or backtest overfitting, see e.g. White (2000) or Bailey et al. (2014, 2017). This bias occurs when more than one strategy is analyzed, which is typically the case. Let us consider 𝑘 portfolio strategies, e.g. three above mentioned approaches to parameters estimation. The out-of-sample performances {𝑝,𝑖=1,…𝑘} are known for these strategies. Of course, only the strategy with the best out-of-sample performance measure 𝑝=max{𝑝,𝑖=1,…𝑘} is usually considered further. However, applying the simple test (16)-(17) would be a mistake as the distribution of a random variable (now the random variable is the performance of the best out of 𝑘 strategies) differs from 𝑃. We denote this new random variable as 𝑃, where 𝑘 is the number of strategies originally analyzed. Note that simple random variable 𝑃 is a special case of 𝑃 for 𝑘=1. Then, the bias-free statistical test should be, 𝑝=Pr [𝑃>max{𝑝,𝑖=1,…𝑘}] (18) for maximization performance measures and 𝑝=Pr [𝑃<max{𝑝,𝑖=1,…𝑘}] (19) for minimization performance measures. 78 Business Systems Research | Vol. 11 No. 2 |2020 The only problem left is the construction of the statistics 𝑃. In order to do so, its meaning must be kept in mind – it is the best performance out of 𝑘-tuples of random portfolios. Thus, a large number of 𝑘-tuples of random weights are generated. The procedure of test statistics calculation is as follows. First, 𝑘-tuples of random numbers 𝑦∈[0,1] from standard uniform distribution are generated and sorted so that 0≤𝑦≤⋯≤𝑦. Then, the simulated weights 𝑥 of the random portfolios are calculated as 𝑥=(𝑦,𝑦−𝑦,𝑦−𝑦,…,𝑦 −𝑦,1−𝑦). (20) This approach guarantees that the sum of weights is equal to one. For each portfolio composition from given 𝑘-tuples the out-of-sample performance is calculated and only the best performance is recorded, i.e. the maximum for maximizing performance measures and minimum for minimizing performance measures. The probability distribution of the statistics is obtained by repeating this procedure many times (in empirical analysis 50,000 times). The last issue, which should be addressed, is the choice of performance measure. The strategies can be evaluated in many ways. The following examples are mentioned: o The annual return corresponds to the out-of-sample return recalculated to annual basis. Investors want to maximize this measure. o The volatility of (daily) returns – the more volatile the returns are, the riskier the investment is. Different measures for the volatility can be applied; the examples are standard deviation (SD), mean absolute deviation (MAD), Value at Risk (VaR), Conditional Value at Risk (CVaR), etc. o Investors usually analyze also maximum drawdown – the maximum relative decline in the portfolio value over the analyzed period, see e.g. Chekhlov et al. (2005) or Magdon-Ismail et al. (2004). o There is also the variety of performance ratios, which are simply the ratios of the reward and risk. The well-known performance ratios are Sharpe ratio (Sharpe, 1966, 1994), Gini ratio (Shalit and Yitzhaki, 1984), mean absolute deviation ratio (Konno and Yamazaki, 1991), mini-max ratio (Young, 1998), Rachev ratio (Biglova et al., 2004) and others. For the summary, see Farinelli et al. (2008). In our paper, the focus is on the portfolio strategies minimizing the risk. Thus, only the performance measures quantifying the riskiness of the portfolio are applied. Results Global Financial Crisis In this section, we study one specific period, which covers the market’s decline phase during the global financial crisis (GFC) to see how efficiently the portfolio strategies worked during the decline of the markets. In order to do so, the period prior to GFC, concretely March 7, 2006 – August 31, 2007, is reserved as the in-sample period. The period with the market’s decline phase is the out-of-sample period (September 1, 2007 – March 2, 2009). The dataset obtained from finance.yahoo.com consists of daily closing prices of the components of Dow Jones Industrial Average index (DJIA). These prices are adjusted for paid dividends and splits. There are 29 stocks included in our analysis and the missing one is the stock of Visa Inc. due to the incomplete data in the chosen period. The dataset covers 3 years, and is evenly split into the in-sample period and the out-of-sample period. In Figure 1, it can be seen that in the in-sample period the DJIA shows an increasing trend, however, in the out-of-sample period, it keeps decreasing due to the 2007-2008 financial crisis. The minimum risk portfolios are 79 Business Systems Research | Vol. 11 No. 2 |2020 calculated from the in-sample data. Then, the verification of the obtained portfolios is performed with the out-of-sample data. Figure 1 Historical Values of DJIA Index in In-sample and Out-of-sample Periods Source: http://finace.yahoo.com Four methods of portfolio strategy creation are analyzed: two models each estimated from daily and weekly returns. The models are Markowitz minimum variance model with classical historical estimation (MA) and fuzzy probabilities minimum variance model (FU). The Bayesian approach is not utilized, because for minimum variance portfolio it suggests the same weights as in case of classical historical estimation. The weights of the portfolios are depicted in Figure 2. Figure 2 Compositions of minimum variance model (MA) and fuzzy probabilities minimum variance (FU) model estimated from daily and weekly returns Source: own calculation As can be seen from Figure 2, which depicts only non-zero weights, the classical historical estimation suggests a more diversified portfolio in terms of the number of assets; however, more than 50% of the portfolio is made up by two stocks (The Coca- 80 Business Systems Research | Vol. 11 No. 2 |2020 Cola Company and Johnson & Johnson). The portfolio of fuzzy probabilities minimum variance model is more evenly diversified, although with fewer assets. For these four portfolios, the performance measures are calculated in the out-ofsample period. Only risk measures are considered. Specifically, the chosen risk measures are maximum drawdown (MDD), which is a commonly applied measure among practitioners, standard deviation of daily returns (SD), which is minimized insample, and mean absolute deviation of daily returns (MAD) as an alternative measure to the standard deviation. The values of the measures for analyzed portfolios together with two commonly applied benchmarks are depicted in Table 1. Table 1 Out-of-Sample Performance Measures with Corresponding P-values Model MDD SD MAD MA – daily data 35.67% (.001) 1.65% (<.0001) 1.05% (<.0001) MA – weekly data 36.57% (.019) 1.64% (<.0001) 1.05% (<.0001) FU – daily data 37.58% (.040) 1.75% (<.0001) 1.12% (<.0001) FU – weekly data 37.95% (.051) 1.75% (<.0001) 1.13% (<.0001) DJIA index 52.25% 2.14% 1.46% 1/N strategy 45.24% 2.23% 1.50% Source: own calculation As can be seen, for all three considered measures, the portfolios perform better than the benchmarks, thus, according to the classical rule of thumb, the conclusion is that these strategies work. However, is the claim that these strategies lower the risk in the out-of-sample period correct from the statistical point of view? To find this out, the distribution of the test statistics must be calculated. In order to do so, 50,000 times 4 (1 respectively) random portfolio compositions are simulated according to (20), for each portfolio the out-of-sample MDD, SD and MAD are calculated, and finally, for every foursome (respectively for each) portfolios the minimums of these risk measures are recorded. In this way, the distributions of the statistics are numerically obtained, see Figure 3. In the figure, the left column corresponds to the simple statistics (17) and the right column depicts the bias-free statistics (19) for four portfolio strategies. The range on the x-axes is the same to make the comparisons easier. Moreover, the fitted Gaussian distributions are added into the graphs. It can be noticed that the bias-free statistics are shifted to the left, which means that for the same observed out-of-sample performance the higher p-value is obtained – there are more randomly generated portfolios with better performances. Moreover, it can be noticed that MDD statistics can be approximated by Gaussian distribution in the case of the simple test. In the case of the bias-free test, the statistics is negatively skewed with a heavier left tail. On the other hand, for SD and MAD the bias-free statics are almost normally distributed, but simple statistics are positively skewed with a heavier right tail. When the mean and standard deviation are known, the classical t-test can be applied. Nevertheless, the p-values for all three bias-free statistics are calculated as the number of random-weights portfolios, which are better than analyzed strategy, divided by 50,000. The calculated p-values are depicted in Table 1. From the results, the following conclusions can be made. Firstly, the portfolio strategies do not create the portfolios randomly (i.e. the null hypothesis is rejected). These strategies lower SD and MAD in the out-of-sample period. Secondly, it can be concluded that the strategy minimizing the variance with standard historical estimation lower the maximum drawdown in the out-of-sample period (p-value for daily and weekly returns 81 Business Systems Research | Vol. 11 No. 2 |2020 is lower than 0.01 and 0.05 respectively). Thirdly, the null hypothesis cannot be rejected for fuzzy probabilities minimum variance model at 1% (5%) significance level for the MDD test and we conclude that this portfolio strategy generates the portfolio composition more or less randomly. To sum it up, the fuzzy probabilities minimum variance model performs slightly worse than Markowitz minimum variance model. Figure 3 Histograms of Risk Measures from 50,000 Randomly Generated Portfolios with Fitted Gaussian Distributions Source: own calculation Rolling Window Tests in Period 2006-2019 In order to prove that the results are robust to the change of the period considered, the tests are performed on a five-year rolling window basis – four years as the in-sample period and one year as the out-of-sample period. The dataset is the daily-adjusted closing prices of the DJIA components in the period from January 3, 2006, to December 31, 2019. There are only 27 stocks included in our analysis as components Dow Inc., NIKE Inc, and Visa Inc. are excluded due to the incomplete data. The source of the data is finance.yahoo.com. The in-sample period consists always of 1,000 daily returns (approximately four years) and the out-of-sample period consists always of 250 daily returns (approximately one year).