In early December the Colorado Business Economic Outlook Forum (CBEO) will be held in Denver for the 46th consecutive year. The event, sponsored by the Leeds School of Business and BBVA Compass is a forecast of state employment based on the expert opinions of estimating groups for each of the NAICS sectors. Each year, one of the most common questions about the event is, “How accurate are the numbers?”
Reasons for Analysis
In the fall of 2008, I presented a simplistic analysis of the accuracy of the CBEO at the annual AUBER conference (this blog post updates that presentation to include the period 1972 to 2010). The original presentation was motivated by curiosity from estimating group members who wanted to know how accurate their forecasts were.
Questions were also raised from an academic perspective.
• Which is more accurate, a forecast calculated by an econometric model or one determined by committees, based on expert opinion?
• Does financial sponsorship of a forecast by a publicly traded financial institution increase the chance of bias in the forecast?
• An academic paper by Owen Lamont (2001) asked, “Does an experienced research team, with a wealth of knowledge, produce a more accurate forecast or does the added knowledge result in an “arrogance” which may reduce the accuracy of the forecast?
While this simplistic analysis provides insight into some aspects of the forecast accuracy, additional research is necessary to address these issues.
Jobs Gained or Lost
The first test of accuracy measured whether jobs were added or lost (+ or -).
Because an employment forecast is intrinsically a growth forecast, its real value occurs if it can identify the years when jobs are lost.
Colorado employment recorded increases in 34 of 39 years.
• The CBEO accurately predicted growth for all 34 years; however, it inaccurately forecasted growth for three years when jobs were lost.
• Said differently, jobs were lost in five years. The CBEO accurately forecasted 2 of the 5 years, or 40%, when job were lost.
Turns (The key is to accurately predict them)
A second test for measuring the accuracy of the CBEO is to evaluate its ability to recognize turns. A turn is defined by a change in the slope or direction of the trend line (upward or downward). For example, it may change directions from positive to zero or negative.
• There were 17 actual turns in 37 years. During those 17 years, the forecast accurately projected turns 12 of 17 times (12 correct turns, 5 incorrect turns). These errors occurred in 1979, 1987, 1990, 1995, and 1998.
The forecast projected 6 turns during the remaining 20 years when turns did not occur (0 correct turns, 6 incorrect turns). These errors occurred in 1973, 1988, 1989, 1993, 1999, and 2009.
• The CBEO accurately forecasted turns 52% of the time – 12 correct/11 incorrect.
This analysis illustrates the difficulty in preparing an employment forecast. There are clearly challenges in accurately predicting turns, as well as the magnitude of each of the turns. This has been demonstrated by most economists in their projections during the Lost Decade. In the case of the CBEO, its value derives more from the discussions associated with projections for each of the sectors, than the projected changes in total employment.
©Copyright 2011 by CBER.
