Treasury's Forecasting Performance: A Head-to-Head Comparison

Goh, Khoon Lek; Lawrence, Daniel Working Paper Treasury's Forecasting Performance: A Head-to-Head Comparison New Zealand Treasury Working Paper, No. 06/10 Provided in Cooperation with: The Treasury, New Zealand Government Suggested Citation: Goh, Khoon Lek; Lawrence, Daniel (2006) : Treasury's Forecasting Performance: A Head-to-Head Comparison, New Zealand Treasury Working Paper, No. 06/10, New Zealand Government, The Treasury, Wellington This Version is available at: https://hdl.handle.net/10419/205586 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/4.0/ Treasury’s Forecasting Performance: A Head-to-Head Comparison Khoon Lek Goh and Daniel Lawrence N EW Z EALAND T REASURY W ORKING P APER 06/10 J ULY 2006 Treasury:875880v2 NZ TREASURY WORKING PAPER 06/10 Treasury’s Forecasting Performance: A Head-to-Head Comparison MONTH / YEAR July 2006 AUTHORS Khoon Lek Goh The Treasury PO Box 3724 Wellington New Zealand Email Telephone Fax [email protected] 04 917 6011 04 499 0992 Daniel Lawrence The Treasury PO Box 3724 Wellington New Zealand Email Telephone Fax [email protected] 04 917 6002 04 499 0992 ACKNOWLEDGEMENTS We would like to thank Satish Ranchhod for supplying us with historical Reserve Bank of New Zealand forecasts. We are also grateful to Yuong Ha and to the participants of Treasury’s Macro Reading Group for providing helpful comments and suggestions. NZ TREASURY New Zealand Treasury PO Box 3724 Wellington 6008 NEW ZEALAND Email Telephone Website [email protected] 64-4-472 2733 www.treasury.govt.nz DISCLAIMER The views, opinions, findings, and conclusions or recommendations expressed in this Working Paper are strictly those of the author(s). They do not necessarily reflect the views of the New Zealand Treasury. The Treasury takes no responsibility for any errors or omissions in, or for the correctness of, the information contained in these working papers. The paper is presented not as policy, but with a view to inform and stimulate wider debate. WP 06/10 Treasury’s Forecasting Performance: A Head-to-Head Comparison i Abstract Work on assessing Treasury’s forecasting performance to date has focussed on comparisons against consensus forecasts. This study compares Treasury’s GDP and CPI forecast performance against individual private sector forecasters as well as major public sector institutions such as the IMF, OECD and the Reserve Bank of New Zealand. The head-to-head comparison makes it possible to assess Treasury’s forecasting performance relative to its peers. When compared across all evaluation periods covering 1996-2005, Treasury’s GDP forecast performance was ranked in the middle at seventh out of 16. The large forecast error for the 1998 year had a material impact on Treasury’s overall forecast performance. Treasury’s CPI forecast performance was not as good, placing tenth out of 12. Large forecast errors for the 1998-2000 period accounted for the poor CPI forecast performance. Treasury’s overall forecast performance was better when evaluating only the current year Budget forecasts, placing fourth for GDP and sixth for CPI. This suggests that Treasury is better at forecasting the current year than the year ahead. Consistent with international studies, no single forecaster consistently outperforms the Consensus, with Treasury beating the Mean 30% of the time for GDP and Consensus 33% of the time for CPI. All forecasters find it difficult to pick recessions and turning points. Large forecasting groups generally have a poorer forecasting record on average. JEL CLASSIFICATION E27 - Forecasting and Simulation E37 - Forecasting and Simulation KEYWORDS Forecast accuracy; New Zealand WP 06/10 Treasury’s Forecasting Performance: A Head-to-Head Comparison ii Table of Contents Abstract...............................................................................................................................i Table of Contents ..............................................................................................................ii List of Tables......................................................................................................................ii List of Figures....................................................................................................................ii 1 Introduction ..............................................................................................................1 2 Data and Methodology.............................................................................................2 2.1 The data .........................................................................................................................2 2.2 The methodology............................................................................................................3 3 Head-to-head Comparison.......................................................................................4 3.1 Forecast performance across all evaluation periods......................................................4 3.2 Forecast performance for Budget current year ..............................................................7 3.3 No one consistently outperforms the Consensus...........................................................9 3.4 Forecasting turning points is difficult ............................................................................10 4 Conclusion..............................................................................................................12 References .......................................................................................................................13 Appendix ..........................................................................................................................14 List of Tables Table 1 – Initial and latest GDP outturn (1996 to 2005)......................................................................2 Table 2 – Forecasters covered in this study.......................................................................................3 Table 3 – Number of observations and forecasters............................................................................3 Table 4 – Average RMSE and relative rank across all evaluation periods.........................................6 Table 5 – Average RMSE and relative rank for Budget current year forecast....................................8 List of Figures Figure 1 – Average relative rank across all evaluation periods..........................................................5 Figure 2 – RMSE across all evaluation periods..................................................................................6 Figure 3 – Average relative rank for Budget current year forecast.....................................................7 Figure 4 – RMSE for Budget current year forecast.............................................................................8 Figure 5 – Frequency of GDP relative ranking over all evaluation periods ........................................9 Figure 6 – Frequency of CPI relative ranking over all evaluation periods ........................................10 Figure 7 – GDP forecast revisions for 1998......................................................................................11 Figure 8 – Treasury GDP forecast revisions and actual...................................................................14 Figure 9 – Treasury CPI forecast revisions and actual.....................................................................14 WP 06/10 Treasury’s Forecasting Performance: A Head-to-Head Comparison 1 Treasury’s Forecasting Performance: A Head-to-Head Comparison 1 Introduction “…the ability to produce accurate predictions of the course of the economy in the near-term future is probably the main criterion by which the public judges the usefulness of our entire profession.” Victor Zarnowitz (December 1986) The Treasury produces forecasts of the New Zealand economy at least twice a year which are published in the Economic and Fiscal Updates.1 The economic forecasts are used as a basis for Treasury’s economic and fiscal policy advice to the Government. Consistently poor forecast performance can lead to policy mistakes, requiring disruptive policy adjustments further down the track. Evaluating Treasury’s forecasting performance is therefore both important and necessary to ensure the quality of its advice. In reality, there are many dimensions to consider when assessing the quality of economic and fiscal advice. Zarnowitz’s statement above, however, provides a simple and high level benchmark for judging one of Treasury’s core functions. Treasury undertakes regular internal monitoring of its forecast performance, and since 2003 its forecasting record has been publicly released on an annual basis.2 Work on comparing Treasury’s forecasting performance to date has mainly focussed on comparisons against consensus forecasts published by the New Zealand Institute of Economic Research (NZIER). International studies have found that consensus forecasts tend to perform better than forecasts produced by individual organisations (Batchelor, 2001; Zarnowitz, 1984; Zarnowitz and Braun, 1992). In this study, a different approach is taken. Treasury’s forecast performance is compared with that of individual private sector forecasters as well as major public sector institutions, rather than simply with the average of other forecasts. This makes it possible to assess how Treasury’s forecast performance compares with its peers, on average and also over time. Section 2 describes the data and methodology used in this paper. Section 3 analyses the head-to-head comparison, and section 4 concludes. 1 These are the Budget Economic and Fiscal Update (published at the time of the Budget, typically in May) and the Half Year Economic and Fiscal Update (published in November or December). In addition, a Pre-election Economic and Fiscal Update is published four to six weeks before a general election. 2 Go to http://www.treasury.govt.nz/forecasts/performance/ for the latest report. WP 06/10 Treasury’s Forecasting Performance: A Head-to-Head Comparison 2 2 Data and Methodology 2.1 The data Forecast data from private sector institutions come from Consensus Economics’ Asia Pacific edition of Consensus Forecasts. Each month, Consensus Economics3 surveys a number of private sector institutions in New Zealand and collects their forecasts for several major economic variables – such as gross domestic product (GDP), private consumption, consumer price index (CPI), unemployment rate and the current account balance. The forecasts are published in the second week of each month, based on the survey conducted in the previous two weeks. Forecast data from major public sector institutions were sourced either directly from those institutions or from published forecasts.4 As well as the forecasts of the individual participants in the survey, Consensus Economics also reports the mean of those forecasts (known as the consensus forecast). In this study, two alternative ‘consensus’ measures are calculated for each period. One is a Mean of the private sector institutions used in this study, plus the Reserve Bank, OECD and IMF. The other is the Median of the forecasts used in the study, calculated to reduce the influence of extreme forecasts. The Mean and Median are calculated only for GDP, since the CPI forecasts used in this study are not on a comparable basis.5 A common dilemma in the forecasting literature is the appropriate “actual outturn” data to use for assessing forecast accuracy. Unlike the CPI, which is generally not subject to revision, initial GDP outturns are often subject to numerous revisions. The revisions could be due to updated information, methodological changes, the introduction of new weights or rebasing (see table 1). Since forecasts made at any point in time are based on all available data at the time (and methodology), and are often judged against the first available data outturn, this study uses the initial outturn as the basis for assessing forecast accuracy. Table 1 – Initial and latest GDP outturn (1996 to 2005) Calendar year Initial outturn Latest outturn %-point difference 1996 2.7 4.1 1.4 1997 2.3 1.9 -0.4 1998 -0.3 -0.1 0.2 1999 3.4 4.4 1.0 2000 3.5 3.4 -0.1 2001 2.4 3.0 0.6 2002 4.4 4.8 0.4 2003 3.5 3.4 -0.1 2004 4.8 4.4 -0.4 2005 2.2 2.2 0.0 High 1.4 Low -0.4 Mean 0.3 3 For more information on Consensus Economics, visit their website at www.consensuseconomics.com. 4 The Reserve Bank provided their forecast data for this study. The OECD’s forecasts were sourced from their twice yearly Economic Outlook publication. The IMF’s forecasts were sourced from their World Economic Outlook reports. 5 Some institutions forecast a different measure of consumer price inflation for part of the evaluation period. See page 4 below. WP 06/10 Treasury’s Forecasting Performance: A Head-to-Head Comparison 3 2.2 The methodology To ensure adequate comparisons, only the Consensus Forecasts which were surveyed in the same month that Treasury’s forecasts were finalised were used, thereby ensuring that the forecasts were based on similar information sets. These are typically the April/May and October/November editions of Consensus Forecasts. Private sector forecasters that are no longer featured in Consensus Forecasts, either because they are no longer included or because they have been merged or taken over by another forecaster, and those without a sufficient number of observations, have been excluded. The Reserve Bank’s forecasts were based on their Monetary Policy Statements which were finalised in the month closest to when the Treasury finalised. The OECD’s Economic Outlook is normally published in June and December, while the IMF’s World Economic Outlook is normally published in May and October. The minor timing differences of when the different forecasts were finalised can have an impact on the forecast performance, given sudden exchange rate or commodity price changes. It is difficult to quantify or resolve the timing differences, and they are an important and ongoing issue for forecast comparisons. The forecast performance of the individual forecasters is not disclosed in this study. They are labelled as either Forecaster X, or Forecasting Group Y. Table 2 below lists the forecasters covered in this study. Table 2 – Forecasters covered in this study Forecasters Forecasting Groups A NZ BERL BNZ IMF Deutsche Bank Infometrics First NZ Capital NZIER UBS OECD Westpac Reserve Bank Due to the limited availability of consistent forecast data, the comparison focuses only on “current year” and “year ahead” forecasts of GDP and CPI, on a calendar year basis (ie for the year ended December) covering the evaluation period 1996 to 2005.6 A current year forecast is defined as one that is made within the calendar year that the forecast period relates to, and a year ahead forecast is one where the forecast is made in the calendar year prior. For example, a forecast made in April 2004 for the 2004 calendar year is a “current year forecast” and the forecast for the 2005 calendar year is a “year ahead forecast”. Table 3 shows the number of observations and forecasters that are included in this study. Note that the number of forecasters includes the Consensus, Mean and Median calculations. Table 3 – Number of observations and forecasters Current year Year ahead Current year Year ahead GDP 308 312 16 16 CPI 227 231 12 12 Number of observations Number of forecasters Forecasts for GDP are all in annual average percent change terms and are comparable across all forecasters except for the OECD, which forecasts on an expenditure GDP 6 The implied forecasting horizons are typically 2, 8, 14 and 20 months ahead. WP 06/10 Treasury’s Forecasting Performance: A Head-to-Head Comparison 4 basis, not production GDP. The forecasts for CPI are not all on a comparable basis. Data from the Consensus Forecasts are for headline CPI in annual average percent change terms, except for a brief period in 1999 and 2000 when it was for CPIX.7 Reserve Bank and Treasury forecasts are for annual percent change of CPIX for the reference years 1996 to 2000, and annual percent change of headline CPI thereafter. In all instances, the appropriate actual outturn was used to calculate the forecast error. The method used to compare forecast performance is similar to the one used by Blix et al (2001). It is based on an average relative rank over all the evaluation periods. For each organisation’s GDP and CPI forecast for an evaluation period, a relative rank is assigned based on the mean absolute error. The most accurate (ie, having the lowest mean absolute error) is given a ranking of 1, the next given a ranking of 2 and so on. An average relative rank is then calculated for each organisation over the entire evaluation period. The average relative rank itself is then ranked to allow for easier comparison. This metric does not put any weightings on good or poor forecasts. For example, two forecasters will have similar rankings if one had two fourth placings and a first, while another had three third placings (both will have average relative rankings of 3). Another metric which places weightings for poor forecast performance is the root mean squared error (RMSE). The RMSE for each forecaster is calculated over all evaluation periods, and ranked. The rankings obtained from the average relative rank based on the mean absolute error can differ, in some cases quite substantially, from that obtained by ranking the RMSE because the latter penalises large forecast errors more severely. Due to the relatively limited sample period, this study focuses more on the average relative rank metric as it does not penalise a forecaster as much for large forecast errors. 3 Head-to-Head Comparison 3.1 Forecast performance across all evaluation periods Figure 1 presents the average relative rank (based on mean absolute error) for each forecaster’s GDP and CPI forecasts across all evaluation periods (April/May and October/November current year and year ahead forecasts for the periods 1996 to 2005). The best forecast performances are those in the lower left of Figure 1 showing the lowest average relative ranks, while the worst forecast performances are in the upper right. Of the 12 forecasters who forecast both GDP and CPI, the best was the Consensus. The Treasury is among the worst performers for CPI forecasts. Figure 2 presents the RMSE of each forecaster’s GDP and CPI forecasts across all evaluation periods. On the RMSE measure, Treasury’s forecast performance is amongst the poorest for both GDP and CPI. Table 4 presents the average relative rank and RMSE for each forecaster across all evaluation periods. For GDP on an average relative rank basis, the Mean was the most accurate followed by the Median, with the Consensus taking third place. Treasury came in close to the middle of the pack at seventh placing out of 16. Treasury came in 13th based on the RMSE. The difference in ranking based on the average relative rank and RMSE highlights how large forecast errors can affect forecast performance. Treasury’s large forecast error for the 1998 period (see Appendix Figure 8) had quite a material 7 CPIX is the Consumers Price Index excluding credit services and interest charges. It was the target measure of inflation for the Reserve Bank until September 1999 when interest charges (but not other credit services) were removed from the CPI. WP 06/10 Treasury’s Forecasting Performance: A Head-to-Head Comparison 11 forecast in October was still for positive growth (albeit small). Loungani notes that the predictive failure could arise either because forecasters lack the requisite information (in terms of reliable real-time data or reliable models) or because they lack the incentives to predict recessions. The only recession period in this study was 1998, and consistent with Loungani’s finding, none of the individual forecasters picked the recession even only a few months out. Figure 7 presents the GDP forecast revisions for calendar 1998. Even by April of that year, most forecasters were expecting reasonable growth with a range of 1.6% - 3.5%, with the Consensus at 2.3%. Treasury at that stage was forecasting growth of 2.4%, close to the Consensus but a large downward revision from the previous forecast of 3.9%. Although subsequent forecasts were revised downwards, it was not until October of that year that most forecasters were predicting a recession. In fact, by August, only 2 of the forecasters covered by the Consensus Forecasts were picking a recession. Treasury at that stage, in a one-off forecast update in August, had revised down its forecast to -0.5%. In the September edition of Consensus Forecasts, almost half of the 13 forecasters surveyed were predicting a recession, but the Consensus remained positive at 0.3%. Figure 7 – GDP forecast revisions for 1998 -1 0 1 2 3 4 5 Jan-97 Apr-97 Jul-97 Oct-97 Jan-98 Apr-98 Jul-98 Oct-98 Consensus Treasury Annual average % change Forecasters find it equally difficult to predict the peak of the economic cycle. The peak of GDP growth over the evaluation period was in 2004 at 4.8%. Twenty months out, the range of forecasts was 2.1% - 3.6%, with the Consensus at 2.7%. Fourteen months out, the forecast range was actually revised downwards to 1.9% - 3.2%, although the Consensus remained at 2.7%. By April 2004, the Consensus was revised up to 3%, but the forecast range of 2.5% - 3.6% still did not encompass the actual outturn. It was not until July that the forecast range included the actual outturn. WP 06/10 Treasury’s Forecasting Performance: A Head-to-Head Comparison 12 4 Conclusion This study extends Treasury’s current forecast accuracy work by making direct comparisons of Treasury’s forecast performance against other forecasters. When compared across all evaluation periods (calendar years 1996-2005), Treasury’s GDP forecast performance was ranked in the middle at seventh place out of 16 on the average relative rank basis. Treasury forecasters misjudged the impact of the Asian financial crisis and droughts on economic activity in 1998, resulting in large forecast errors at that time which had a material impact on Treasury’s overall forecast performance. This is reflected in a relatively large RMSE compared to other forecasters. When compared only against other individual forecasters across all evaluation periods (ie, excluding Consensus, Median and Mean), Treasury’s GDP forecast performance is in the top half, and better than other major forecasting groups. However, Treasury’s CPI forecast performance ranks in the bottom half, out-performing only two other forecasting groups. Large forecast errors for CPI relating to the 1998-2000 period were a large contributor to the poor forecast performance. Treasury’s forecast performance was better when evaluating only the current year Budget forecasts, placing fourth for GDP and sixth for CPI. This suggests that Treasury is better at forecasting the current year than the year ahead. Compared to other individual forecasters, Treasury’s Budget current year GDP forecast was bettered only by one other forecaster. Consistent with international studies, no single forecaster consistently outperforms the Consensus, and all forecasters find it difficult to pick recessions and turning points. Large forecasting groups, on average, tend to perform poorly at forecasting GDP compared to private sector forecasters on the average relative rank basis. This finding is similar to Blix et al (2001). But their performance is comparable on the RMSE ranking, suggesting that forecasting groups tend to be more conservative in their forecasts, which may not result in a closer forecast to the actual, but lessens the chance of a large forecast error. Another possible explanation is that major forecasting groups tend to produce comprehensive forecasts at certain times during the year, which do not coincide with the dates chosen for this study. Their responses to Consensus Economics could therefore be quick updates which reduce their forecast performance. It is difficult to quantify and resolve this timing issue which may affect forecasting performance as measured in this comparison. WP 06/10 Treasury’s Forecasting Performance: A Head-to-Head Comparison 13 References Batchelor, R. (2001) How useful are the forecasts of intergovernmental agencies? The IMF and OECD versus the consensus, Applied Economics 33, p225-235. Blix, M., Wadefjord, J., Wienecke, U. and Adahl, M. (2001) How good is the forecasting performance of major institutions?, Economic Review 3, p38-68. Juhn, G. and Loungani, P. (2002) Further cross-country evidence on the accuracy of the private sector’s output forecasts, IMF Staff Papers, Vol. 49, No. 1, p49-64. Lenain, P. (2001) What is the track record of OECD economic projections?, OECD note. Loungani, P. (2001) How accurate are private sector forecasts? Cross-country evidence from consensus forecasts for output growth, International Journal of Forecasting 17, p419- 432. Zarnowitz, V. (1986), "The Record and Improvability of Economic Forecasting", National Bureau of Economic Research Working Paper 2099. Zarnowitz, V. (1984), "The accuracy of individual and group forecasts from business outlook surveys", Journal of Forecasting, 3(1), 11-26. Zarnowitz, V. and Braun, P. (1992), “Twenty-two year of the NBER-ASA quarterly economic outlook surveys: aspects and comparisons of forecasting performance”, National Bureau of Economic Research Working Paper 3965. WP 06/10 Treasury’s Forecasting Performance: A Head-to-Head Comparison 14 Appendix Figure 8 – Treasury GDP forecast revisions and actual -1 0 1 2 3 4 5 6 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 Actual Treasury forecasts A nnual average % change Figure 9 – Treasury CPI forecast revisions and actual 0.0 1.0 2.0 3.0 4.0 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 Actual Treasury forecasts A nnual % change