Error correction (or blame deflection?)

For anyone interested in the practice and methodological issues associated with economic forecasting, you could do a lot worse than read the Bernanke Review of forecasting at the Bank of England. According to the FT, the former Fed chair who was commissioned to produce a report on the BoE’s forecasting practices after its failure to predict the rise in inflation in 2022, was “brutally honest about [its] failings.“ Brutal might be overstating it, but it was an honest assessment that one feels is shared by many BoE insiders. As one might expect, not all economists agreed with all of its conclusions but there was a lot to like about it.

The fact that Bernanke outlined many shortcomings in the BoE’s practices should come as no surprise. No system is ever perfect, and the fact that the current monetary framework has been in place for almost 30 years does suggest that it is time to have a close look. There are a number of questions around the whole process, however. Why was it necessary to have such a review in the first place? If the processes really are as poor as Bernanke highlighted, why did it require an external review to point it out? And if the purpose of the exercise was to address policy errors, should we not be spending time looking at the policy making process rather than putting a lot of effort into the forecast generation process? I will deal with these points below.

What were the conclusions?

It is perhaps instructive first to reflect on Bernanke’s main process recommendations. One of the most widely trailed in advance was the suggestion that the BoE publish scenarios alongside the main forecast. This would “help assess the costs of potential risks to the outlook” and “stress test the judgements made by the MPC.” There is a lot of merit in doing this: The experience of recent years which has produced the Covid-19 pandemic and the oil price shock, suggests that a single forecast with a univariate central case cannot adequately capture all future states of the world. Even allowing for risks in the form of a fan chart, no forecast could capture shocks of the magnitude of 2020 (see chart below). The Bernanke Review went as far as suggesting that “the fan charts as published in the MPR have weak conceptual foundations, convey little useful information over and above what could be communicated in other, more direct ways, and receive little attention from the public. They should be eliminated.” While there is some truth in this, it may be going too far to eliminate them, as fan charts are a very useful way of conveying risks around a central case in a stable environment, and there is a case for retaining them.

Another very important consideration was the nature of conditioning assumptions, particularly for the future path of interest rates. There are a number of reasons why using market rate expectations as the appropriate starting point is less than optimal. For one thing, “forward rates implied by the market curve are not pure forecasts of future rates, because forward rates may incorporate risk and liquidity premiums.” In addition they may not reflect the MPC’s best judgement of the path of rates, meaning that “a forecast conditioned on the market curve may be misleading.” One alternative is for the central bank to give a preferred path for rates, much as the Riksbank does, although as I argued in this post in 2019, this could simply create a hostage to fortune. Instead, the practice of offering alternative scenarios based on different rate paths will probably suffice.

A final big point, and one that is close to my heart, is that the software required to manage and manipulate data “is seriously out of date and difficult to use” and should be upgraded and constantly monitored. I don’t know which systems Bernanke is referring to but my own experience with languages such as R and Python, now en vogue in economic circles, is that they are far less user-friendly and flexible than some of the systems designed in the 1970s. The review was also critical of the BoE’s macro model, COMPASS, unveiled to great fanfare in 2013. Bernanke did not explicitly say that DSGE models may not be up to the job of forecasting but he offered the view that structural models (of the kind I have long advocated) still have a role to play in forecasting – after all, the Fed still uses them.

Policy considerations

The elephant in the room, however, is why it was felt that such a review was required in the first place. The answer, to put it bluntly, is that it was designed to keep politicians off the BoE’s back after it was accused of failing to predict the huge rise in inflation in 2022 (true) and the fact that its policy response was too slow (less true). In fact, the BoE's inflation forecast in February 2022 was above that of the consensus, predicting end-2022 inflation at 5.8% versus a consensus expectation of 4.6% (outturn: 10.8%) and end-2023 inflation at 2.5% versus the consensus prediction of 2.1% (outturn: 4.2%). Thus, while the BoE forecast was a significant under-estimate, it was less so than most forecasters.

As for the policy response, as I (and many others) have noted previously there was little anyone could have done to prevent an inflation spike in the face of an external oil price shock. Recall that the UK had just come off the back of a pandemic which had resulted in the steepest decline in output in 300 years and whose long-term effects were at that time still unknown. It did not feel like the right time for a sharp tightening of monetary policy. However a review of process is a standard response to issues that are more a matter of policy. Simply put, it is a way to deflect attention.

Another issue worth addressing is the question raised by the Sunday Times economics editor David Smith as to why it took an external review to highlight these shortcomings, which were well known internally. We are very much in speculative territory here, but since Bernanke took a lot of evidence from BoE insiders – past and present – it is hard to avoid the conclusion that this review offered an opportunity to tackle internal inertia. This may be the result of senior managers lack of knowledge of the issues involved; the fact that their attention has been diverted by other policy matters in recent years (Brexit, the pandemic) or simply a lack of budget resources. Either way the Review is a good way to get their attention.

Last word

It is always a good thing to review forecast models and processes, especially when they have been in place for so long and the Bernanke Review put the BoE’s process under a lot of scrutiny. In many ways it simply came across as a call to modernise a system, which in the grand scheme of things was already pretty decent but perhaps had been neglected a little over the past decade. However, the one thing it will not fix is that the future is inherently unknowable. No matter how state of the art, no forecasting system can cope with the kind of shocks to which we have been subject of late. Give it another decade and we will be having this debate all over again.