How well did economic models perform in the COVID-19 economic crisis?

A Covid-19 closure signboard.

This article compares forecasts from 2020 with actual outcomes. You can view the latest economic forecasts from the Cambridge Centre for Risk Studies by visiting the Cambridge Business Risk Hub.

By Dr Scott Kelly, Senior Risk Researcher, Cambridge Centre for Risk Studies

At the Cambridge Centre for Risk Studies, we estimate the consequences of extreme – and often unlikely – scenarios. We use a number of different types of models to explore “what would happen if…” Macroeconomic models are a key class of models that we employ, and often we push them outside their comfort zone in the scenarios we construct and analyze.

Tracking economic projections during the pandemic

The Covid-19 crisis that unfolded was a real-world test of macroeconomic forecasting. At CCRS we had published a hypothetical pandemic scenario in 2015, and used macroeconomic models to estimate the likely consequences on global economic metrics. So we followed the macroeconomic forecasts of the Covid-19 pandemic with great interest. Many organizations made projections of what they estimated the economic consequences would be of the pandemic and government responses. We documented these forecasts, and presented our compilations in our Covid-19 Webinars in 2020. We plotted the economic forecasts for nineteen of the world’s largest economies, collectively accounting for 80% of global GDP, on our CambridgeBusinessRiskHub and bracketed the range of opinions as a range of scenarios, L1 to L4 for each country, for use by businesses in their contingency planning.

Analysis of how models performed

Twelve months after these initial forecasts were published, we can start to unpack what exactly happened and how well the experts and their models performed in predicting the actual outcomes. This will help us model future events – of pandemics and many other types of emerging risks – better in the future.

Table 1 shows the prevailing economic forecasts of GDP for the 19 countries as they were published in May 2020, September 2020 and February 2021, and compares them with the actual economic output figures for Q4 2020.

Forecasts were pessimistic

The first observation is that the economies generally performed better than the models expected them to. Early in the pandemic, economic forecasts painted a dire picture of the economic fallout that was to come. A second observation is that the range of scenario variants, L1 to L4, adequately bracketed the range of outcomes that did occur for all but two of the 19 economies. Companies that used the scenario variants for contingency planning around the range of potential outcomes were not caught out by the actuality.

Our analysis, presented below, shows that most economies did not collapse for as long or as deep as predicted by many of the leading economic models. While some countries fared worse than others, the majority of countries were far more resilient to the shock of lockdowns and future uncertainty than most economists had predicted. In fact, most countries have now surpassed pre-pandemic GDP levels with many countries even surpassing their pre-pandemic forecast GDP levels. In other words, GDP for these countries is now higher than it would have been at this point had there been no pandemic at all. This is remarkable considering the extent and duration of lockdowns that many countries have endured over the last 12 months.

Government stimulus packages

One reason for the underestimation is that the economists failed to account for the scale of stimulus packages that national governments injected into their economies in unprecedented interventions with deficit spending.

There are two possible interpretations. Either markets were already operating efficiently prior to the pandemic, and this new injection of spending has overstimulated the economy beyond its potential output, meaning that runaway inflation will follow and a rapid increase in official interest rates is likely over the medium term. Or markets prior to the pandemic were imperfect, and the macroeconomic shock and government spending improved macroeconomic efficiency, killing-off poor performing firms and allocating capital to more productive uses. The second, more optimistic outlook suggests that the pandemic has been an important catalyst for kick-starting the economy into a new healthier state.

Using scenarios for preparedness planning

As with all uncertainty we urge that businesses are prepared for the range of potential outcomes that could be consistent with either interpretation of the facts. Have contingency plans for the possibility of an inflationary period, as well as plans to exploit a new disruptive wave of economic creativity that has been set in motion.

Our approach suggests that a scenario risk-based approach is better than following a best-estimate projection. Scenarios are able to place an envelope around possible outcomes and are useful for preparedness, contingency planning, communicating assumptions and uncertainty, and exploring options for what-ifs. Businesses using a range of scenarios are less likely to find themselves surprised by projection failures. In decision science this is known as minimizing regret in planning.

Scenarios enable companies to minimize regret in their crisis management.

Analysis of Economic Forecasts

In the following section we present the prevailing economic forecasts as they were published in May 2020, September 2020 and February 2021. By returning to analyse the economic forecasts post-hoc we are able to test the efficacy of economic models and their ability to accurately forecast economic output after a significant economic shock. This type of post-hoc analysis is seldom done in practice but instructive for the purpose of understanding the limitations of using economic models during an unfolding crisis. As well as comparing and summarising the economic forecasts from various published reports and papers, we also undertook our own economic projections using the Oxford Economic model for nineteen of the world’s largest economies. These economies collectively represent 80% of global GDP and include both developed and developing countries (See Table 1).

Table 1 compares the forecast economic output for second quarter of 2021 for 19 different countries. Economic forecasts were taken in May 2020, September 2020 and February 2021. What’s clear from this table is that overall economic output has improved for the majority of countries as the pandemic continued to evolve over 2020. Only in India, Russia, Brazil and Indonesia have the economic forecasts worsened as the pandemic continued. Similarly, the economy has outperformed pre-Covid forecasts in Switzerland, Netherlands, South Korea, Italy, France, Australia, Germany and China. The remaining group of countries which includes Turkey, Mexico, UK, Spain, Canada, Japan and the United States have worse pre-pandemic forecasts for the second quarter in 2021.

Table 1: Economic Model Projections from May 2020 to February 2021 ($US Billion)

When trying to explain the differences in model forecasts as shown in Table 1, the results do not only correlate with the health impacts from the pandemic. The differences in output appear to more closely correlate with the level of stimulus provided by governments. As shown in Figure 1 the countries with the highest stimulus are also those have performed better than forecast.

Figure 1: Government Fiscal Stimulus per person as of December 2020.

To show just how quickly economic models were adjusting to the unfolding events of the pandemic, Figure 1 shows each of these three forecasts for the United States. In May 2020 most economic models were predicting a deep and long recession, but by September, even as the Pandemic had not peaked in the United States, economic models were predicting much more optimistic outcomes. By February this year, the same models were predicting an even better rebound with output essentially back to pre-covid levels by the fourth quarter of 2021.

Figure 2: US Pandemic Projections.

Given the inherent uncertainty of crises and the proven unreliability of well-established macroeconomic models,  the Cambridge Centre for Risk Studies proposes a new approach to forecasting economic output during a crisis. This new approach adopts a scenario risk based approach that captures the envelope of possible future scenarios that vary probabilistically. Using this method, each economic forecast represents a scenario and is assigned a probability weighting. Probabilities for scenarios can be estimated from historical precedent, expert judgement, secondary economic models or statistical analysis. The total number of scenarios considered is not important, but we recommend using at least three. Once each economic forecast has been assigned a probability weighting (where the sum of all forecast probabilities is equal to one) the set of forecasts can be combined to represent a single risk weighted probability density function of economic forecasts.

At the Cambridge Centre for Risk Studies our typical scenario analysis includes four variants of a scenario, these are labelled L1 to L4 from least to most impactful. The most impactful scenario, L4,  has the lowest likelihood of occurrence, with likelihoods for our pandemic modelling calibrated to those in Table 2 below. As shown here, L2 has the highest probability of occurrence but this is bounded by a distribution of scenarios captured by the envelope represented by the probability distribution from L1 to L4. Using the L1-L4 scenario framework, four economic forecasts were created for each of the 19 countries. These forecasts are also shown graphically in Table 2 below.

Table 2: Scenario probabilities.
Figure 3: Risk based scenarios for macroeconomic projections.


Conclusion

In this blog we have shown that existing macroeconomic models are poor at predicting economic output after a crisis. As the pandemic unfolded in 2020, macroeconomic forecasts were continually updated as new information came to hand. Although not the only factor, government stimulus appears to be driving factor behind countries performing better than forecast. Projecting economic output after and during a crisis is difficult using existing macroeconomic models tuned to operate during normal periods. In order to overcome these shortcomings a risk-based scenarios approach is recommended.

Cambridge Centre for Risk Studies

Cambridge Centre for Risk Studies

Cambridge Centre for Risk Studies

Latest posts by Cambridge Centre for Risk Studies (see all)

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.