Forecast error

In statistics, a forecast error is the difference between the actual or real and the predicted or forecast value of a time series or any other phenomenon of interest. Since the forecast error is derived from the same scale of data, comparisons between the forecast errors of different series can only be made when the series are on the same scale.

In simple cases, a forecast is compared with an outcome at a single time-point and a summary of forecast errors is constructed over a collection of such time-points. Here the forecast may be assessed using the difference or using a proportional error. By convention, the error is defined using the value of the outcome minus the value of the forecast.

In other cases, a forecast may consist of predicted values over a number of lead-times; in this case an assessment of forecast error may need to consider more general ways of assessing the match between the time-profiles of the forecast and the outcome. If a main application of the forecast is to predict when certain thresholds will be crossed, one possible way of assessing the forecast is to use the timing-error—the difference in time between when the outcome crosses the threshold and when the forecast does so. When there is interest in the maximum value being reached, assessment of forecasts can be done using any of:

• the difference of times of the peaks;
• the difference in the peak values in the forecast and outcome;
• the difference between the peak value of the outcome and the value forecast for that time point.

Forecast error can be a calendar forecast error or a cross-sectional forecast error, when we want to summarize the forecast error over a group of units. If we observe the average forecast error for a time-series of forecasts for the same product or phenomenon, then we call this a calendar forecast error or time-series forecast error. If we observe this for multiple products for the same period, then this is a cross-sectional performance error. Reference class forecasting has been developed to reduce forecast error. Combining forecasts has also been shown to reduce forecast error.

Calculating forecast error

The forecast error is the difference between the observed value and its forecast based on all previous observations. If the error is denoted as $e(t)$ then the forecast error can be written as:

$e(t)=y(t)-{\hat {y}}(t|t-1)$

where,

$y(t)$ = observation

${\hat {y}}(t|t-1)$ = denote the forecast of $y(t)$ based on all previous observations

Forecast errors can be evaluated using a variety of methods namely mean percentage error, root mean squared error, mean absolute percentage error, mean squared error. Other methods include tracking signal and forecast bias.

For forecast errors on training data

$y(t)$ denotes the observation and ${\hat {y}}(t|t-1)$ is the forecast

For forecast errors on test data

$y(t+h)$ denotes the actual value of the h-step observation and the forecast is denoted as ${\hat {y}}(t+h|t)$

Academic literature

Dreman and Berry in 1995 "Financial Analysts Journal", argued that securities analysts' forecasts are too optimistic, and that the investment community relies too heavily on their forecasts. However, this was countered by Lawrence D. Brown in 1996 and then again in 1997 who argued that the analysts are generally more accurate than those of "naive or sophisticated time-series models" nor have the errors been increasing over time.

Hiromichi Tamura in 2002 argued that herd-to-consensus analysts not only submit their earnings estimates that end up being close to the consensus but that their personalities strongly affect these estimates.

Examples of forecasting errors

Michael Fish - A few hours before the Great Storm of 1987 broke, on 15 October 1987, he said during a forecast: "Earlier on today, apparently, a woman rang the BBC and said she heard there was a hurricane on the way. Well, if you're watching, don't worry, there isn't!". The storm was the worst to hit South East England for three centuries, causing record damage and killing 19 people.

Great Recession - The financial and economic "Great Recession" that erupted in 2007—arguably the worst since the Great Depression of the 1930s—was not foreseen by most forecasters, though a number of analysts had been predicting it for some time (for example, Brooksley Born, Dean Baker, Marc Faber, Fred Harrison, Raghuram Rajan, Stephen Roach, Nouriel Roubini, Peter Schiff, Gary Shilling, Robert Shiller, William White, and Meredith Whitney). The UK's Queen Elizabeth herself asked why had “nobody” noticed that the credit crunch was on its way, and a group of economists—experts from business, the City, its regulators, academia, and government—tried to explain in a letter.

It was not just forecasting the Great Recession, but also its impact where it was clear that economists struggled. For example, in Singapore, Citi argued the country would experience "the most severe recession in Singapore’s history". The economy grew in 2009 by 3.1%, and in 2010 the nation saw a 15.2% growth rate.

Similarly, Nouriel Roubini predicted in January 2009 that oil prices would stay below $40 for all of 2009. By the end of 2009, however, oil prices were at $80. In March 2009, he predicted the S&P 500 would fall below 600 that year, and possibly plummet to 200. It closed at over 1,115 however, up 24%, the largest single-year gain since 2003. CNBC's Jim Cramer wrote that Roubini was "intoxicated" with his own "prescience and vision," and should realize that things are better than he predicted; Roubini called Cramer a "buffoon," and told him to "just shut up". Although in April 2009, Roubini prophesied that the United States economy would decline in the final two quarters of 2009, and that the US economy would increase just 0.5% to 1% in 2010, in fact the U.S. economy in each of those six quarters increased at a 2.5% average annual rate. Then in June 2009 he predicted that what he called a "perfect storm" was just around the corner, but no such perfect storm ever appeared. In 2009 he also predicted that the US government would take over and nationalize a number of large banks; it did not happen. In October 2009 he predicted that the price of gold "can go above $1,000, but it can’t move up 20-30%”; he was wrong, as the price of gold rose over the next 18 months, breaking through the $1,000 barrier to over $1,400.

2020 Global Growth - At the end of 2019 the International Monetary Fund estimated global growth in 2020 to reach 3.4%, but as a result of the coronavirus pandemic, the IMF have revised its estimate in November 2020 to expect the global economy to shrink by 4.4%.

See also

• Calculating demand forecast accuracy
• Errors and residuals in statistics
• Forecasting
• Forecasting accuracy
• Mean squared prediction error
• Optimism bias
• Reference class forecasting