ARCH Models for Multi-period Forecast Uncertainty - University at

54 Pages·2009·0.32 MB·English

Published in Econometric Analysis of Financial
and Economic Time Series – Part A (Eds. D. Terrell
and T.B. Fomby), Elsevier, JAI.

ARCH Models for Multi-period Forecast Uncertainty
– A Reality Check Using a Panel of Density Forecasts

Kajal Lahiri and Fushang Liu

Department of Economics, University at Albany – SUNY
Albany, NY 12222. USA. Email: [email protected]

Abstract
We develop a theoretical model to compare forecast uncertainty estimated from time
series models to those available from survey density forecasts. The sum of the average
variance of individual densities and the disagreement is shown to approximate the
predictive uncertainty from well-specified time series models when the variance of the
aggregate shocks is relatively small compared to that of the idiosyncratic shocks. Due to
grouping error problems and compositional heterogeneity in the panel, individual
densities are used to estimate aggregate forecast uncertainty. During periods of regime
change and structural break, ARCH estimates tend to diverge from survey measures.

Keywords: Inflation, Survey of Professional Forecasters, GARCH, Real time data,
Disagreement.

JEL: E31; E37; C23; C53

1
1. Introduction

Macroeconomic uncertainty as characterized by aggregate forecast density or intervals
has long been recognized as an important factor in determining economic outcomes.1
Milton Friedman, in his Nobel lecture (1977), conjectured that high inflation causes high
inflation uncertainty, and that high uncertainty in turn causes inefficiencies in the
economy. The latest economics Nobel laureate Robert Engle (1983) wrote while
introducing his celebrated ARCH model,
“When inflation is unpredictable, risk averse economic agents will incur loss,
even if prices and quantities are perfectly flexible in all markets. Inflation is a
measure of the relative price of goods today and goods tomorrow; thus,
uncertainty in tomorrow’s price impairs the efficiency of today’s allocation
decisions.”
Not surprisingly, these developments have triggered much research studying the effects
of inflation uncertainty on savings, interest rates, investment, and other variables. More
recently, the introduction of inflation targeting regimes in U.K., Canada, Sweden, and
many other countries during the 1990s has rekindled an enormous interest in generating
credible inflation forecasts and the uncertainty associated with the forecasts.2

In order to deal with the causes and consequences of forecast uncertainty in a convincing
way, it has to be measured correctly. This is particularly important for predictive
uncertainty because, unlike forecasts, variance of predictive errors generated by macro
time series models is always unobserved, and hence, can never be evaluated against
subsequently observed realizations. During last twenty years, the most popular approach
to estimate forecast uncertainty has been the univariate or multivariate ARCH models
initiated by Engle (1982, 1983), who modeled the conditional variance of inflation
forecast error as an autoregressive conditional heteroskedastic (ARCH) process.
Bollerslev (1986), Nelson (1991), Glosten et al. (1993) and others have generalized the

1 Interval and probability forecasts have a longer track record in weather forecasts and sports picks.
2 See, for, instance, Sims (2002), Šmidková (2003), Cogley et al. (2003), and Garratt et al. (2003).
2
basic ARCH model in various directions, and have found many applications in finance
and economics.

There are, however, several possible reasons why ARCH-type models may not produce a
reliable proxy for inflation uncertainty. First, the reliability of this measure depends on
the reliability of the conditional mean and variance functions. This is why Engle (1982,
1983) had emphasized the need for various specification tests in these models. More
generally, Rich and Tracy (2003) have questioned the use of conditional variance of a
series which is essentially related to its ex post predictability as a measure of forecast
uncertainty which is related to ex ante confidence of a prediction. Second, most ARCH-
type models assume that the regime characterizing inflation and inflation uncertainty is
invariant over time. This assumption may not be true due to structural breaks. Although
some researchers have tried to rectify this deficiency (see, for example, Evans (1991),
Evans and Wachtel (1993)), the success of these efforts relies on correctly specifying the
structural break itself. Third, as noted by Bomberger (1996) and McNees (1989),
forecasters use different models. Thus, uncertainty regarding the model that generates
inflation may be a large part of inflation forecast uncertainty. Conditional variance
generated by a single model cannot capture this important feature. Finally, Zarnowitz and
Lambros (1987) observed that forecast uncertainty depends not only on past forecast
errors but also on many future-oriented factors such as policy changes, macro shocks,
data revisions and many others including purely subjective factors. ARCH-type models
ignore this forward-looking behavior of forecasters and other relevant factors that are not
included in the time series specification.3

In the U.S., research on the measurement of forecast uncertainty was facilitated by the
foresight of Victor Zarnowitz who under the aegis of American Statistical Association
and the National Bureau of Economic Research pioneered a survey in 1968 in which,
apart from other point forecasts, subjective density forecasts of real GDP and inflation

3 Sims (2002) discusses the current state and limitations of models generating macro forecasts and their
uncertainty. Lahiri and Liu (2005) have examined the validity of the ARCH-type specifications using data
on density forecasts.
3
were elicited from the survey respondents.4 The average variance of the individual
probability distributions can then be directly used as a time series measure of forecast
uncertainty. Studies that used this measure include, among others, Zarnowitz and
Lambros (1987), Lahiri and Teigland (1987), Lahiri, Teigland and Zaporowski (1988)
and Batchelor and Dua (1993). This measure is attractive because the variance associated
with the density forecast truly represents the uncertainty perceived by an individual
forecaster. Batchelor and Dua (1995) and Giordani and Soderlind (2003) argued that this
measure reflects the uncertainty a reader of survey forecasts faces if he randomly picks
and trusts one of the point forecasts. One question here is whether we should also include
forecast disagreement across individuals as a component of aggregate uncertainty. Since
forecast disagreement reflects people’s uncertainty about models, arguably it should be a
part of the aggregate forecast uncertainty.

Due to its ready availability, the variance of the point forecasts (or disagreement) of the
survey respondents has been widely used as a proxy for inflation uncertainty. Cukierman
and Wachtel (1979, 1982) investigated the relationship between inflation uncertainty and
inflation level as well as the variance of the change of relative prices. Although forecast
disagreement is easy to compute, its disadvantages are obvious. First, individual biases
may be part of forecast disagreement and not necessarily reflect forecast uncertainty.
Secondly, as noted by Zarnowitz and Lambros (1987), it is possible that individual
forecasters are extremely certain about their own forecasts, yet the forecasts themselves
are substantially dispersed. Conversely, the uncertainty of individual forecasts may be
high but point forecasts may be close. Disagreement would then overstate uncertainty in
the former case, and understate it in the latter.

There is rather a large literature comparing these three measures of predictive confidence.
Engle (1983) was the first to discuss the relationship between the forecast uncertainty
derived from the ARCH model and from survey data. He showed that the conditional
variance is approximately equal to the average individual forecast error variance.

4 Since 1990 the Federal Reserve Bank of Philadelphia manages the survey, now called the Survey of
Professional Forecasters (SPF).
4
Zarnowitz and Lambros (1987) found that although the average forecast error variance
and forecast disagreement were positively correlated, the latter tends to be smaller than
the former. Batchelor and Dua (1993,1996) compared the average individual forecast
error variance with a number of proxies including forecast standard deviations from
ARIMA, ARCH and structural models of inflation. They found that these proxies are not
significantly correlated with the average individual forecast error variance. Other related
works include Bomberger (1996), Evans (1991) and Giordani and Soderlind (2003).
However, one drawback common to most of these studies is that when comparing
different measures of inflation uncertainty the supporting information sets were often not
exactly the same. For example, forecasters in a survey often have more information than
historical data. They have partial information about current period when making
forecasts. So a direct comparison of forecast disagreement or average individual forecast
variance with a time series measures may not be appropriate. Second, due to
heterogeneity in forecasts, survey measures often have individual biases. When
comparing survey measures with time series estimates, we should first correct for these
compositional effects in the panel. Finally, most existing studies are empirical. We need
a theoretical framework to compare estimates of uncertainty based on aggregate time
series and survey data on forecast densities.

In this paper, we propose a simple model to characterize the data generating process in
SPF. We argue that the estimated variance of the aggregate density as proposed by
Diebold et al. (1999) and Wallis (2004) gives a reasonably close lower bound on the
forecast uncertainty imbedded in a well-specified time series model. Since the former
variance is simply the sum of average variance of the individual densities and the forecast
disagreement (cf. Lahiri et al. (1988)), our model presents a way of justifying the use of
disagreement as a component of macroeconomic uncertainty.5 Considering that time
series models often have misspecification problems, the sum of forecast disagreement
and average individual forecast error variance from surveys may provide a dependable
proxy for the aggregate forecast uncertainty over diverse periods. Our empirical analysis

5In recent research, forecast heterogeneity has appeared prominently as a component of overall
macroeconomic uncertainty. See, for instance, Kurz (2002), Carroll (2003), Mankiw et al. (2003), Souleles
(2004), and Giordani and Söderlind (2005).
5
shows that the time series and survey measures are remarkably similar during periods of
low and stable inflation. But during periods of high and unpredictable inflation, the two
tend to diverge.

The remainder of the paper is organized as follows. In section 2, we introduce our
theoretical model to compare time series and survey measures of inflation uncertainty. In
section 3, we show how to extract the measure of forecast uncertainty from the SPF data
as defined in section 2. Sections 4 and 5 describe the SPF forecast density data, and
present estimates of inflation forecast uncertainty defined as the sum of average
individual forecast error variance and forecast disagreement. In sections 6 and 7, we
empirically compare the survey measures estimated in section 5 with those estimated
from some popular ARCH models. Section 8 concludes the paper.

2. The model

In this section, we propose a model to study the relationship between measures of
forecast uncertainty based on survey data and time series models. Giordani and Soderlind
(2003) and Wallis (2004) have discussed how a panel of density forecasts should be used
to generate forecast uncertainty representing the entire economy. However, they did not
try to find the relationship between survey and time series measures of forecast
uncertainty.

Consider the prediction of the log of current general price level, p , the actual of which is
t
available only in period t+1. Suppose the information set of forecaster z is I (z). Then
t
the prediction of p made by forecaster z can be expressed asE(p | I (z)). In period t,
t t t
individual forecasters can obtain information from two sources. First, they have
knowledge of the past history of the economy in terms of its macroeconomic aggregates.
This aggregate information may, for instance, be collected from government publications.
Often macro forecasts are generated and made publicly available by government agencies
and also by private companies. We assume that this information is common to all
6
forecasters as public information.6 Although public information does not permit exact
inference of p , it does determine a “prior” distribution of p , common to all forecasters.
t t
We assume the distribution of p conditional on the public information
t
is p ~ N(p ,σ2). Secondly, since SPF forecasters usually make forecasts in the middle
t t
of the quarter for the current and the next year, they will invariably have some private
information about p . Thus, while reporting the density forecasts, they have partial
t
information about the first half of period t although official report of p would still not
t
be available. This private information may take the form of data on relevant variables that
are available at higher frequencies (e.g., CPI, unemployment, Blue Chip monthly
forecasts, etc.) that individual forecasters may use and interpret at their discretions. More
importantly, it will be conditioned by individual forecasters’ experience and expertise in
specific markets. A key feature of this private information is that it is a mixture of useful
information about p and some other idiosyncratic information that is orthogonal to p .7
t t
If we use p (z) to denote the private information of forecaster z about p , then
t t
p (z)will be a function of p and the idiosyncratic factor denoted also by z with
t t
z ~ N(0,τ2). For simplicity, we assume
z
p (z) = p + z (2-1)
t t
By definition, z is distributed independent of p and individual-specific information of
t
other forecasters. Although forecaster z knows p (z), he is not sure about the relative
t
size of p and z. The problem for the individual is how to infer about p from the public
t t
information and the private information, p (z). This is a typical signal extraction
t
problem.8

6 For example, it is commonly believed that the USDA forecasts for agricultural prices are used as
‘benchmark” by many private and public forecasters, see Irwin et al. (1994) and Kastens et al. (1998).
7 Our formulation is also consistent with the evidence presented by Furher (1988) that survey data contain
useful information not present in the standard macroeconomic database.

8 In Lucas’ signal extraction model (1972, 1973), this private information is assumed to be the price in
market z. Kurz and Motolese (2001) use a similar decomposition.
7
Using (2-1) it is easy to show that the distribution of p conditional on both the public
t
and private information is normal with mean
E(p | I (z)) = E(p | p (z),p ) = (1−θ )p (z)+θ p (2-2)
t t t t t z t z t
and variance θσ2, where θ =τ2 /(τ2 +σ2).
z z z z

Consider an outside observer (or a central policy maker) who is to forecast the aggregate
price level. The information available to him is only public information. He has no
private information about period t that can help him to better infer about p . However, an
t
unbiased point forecast of p can be made with available public information. Under
t
mean squared error loss function, the optimal point forecast should be p , with forecast
t
uncertainty being σ2. Note that this is just the forecast and forecast uncertainty from
aggregate time series models if the mean and the variance equations were correctly
specified.

The situation facing forecaster z is different. He has both the public and private
information. Given the information set I (z), his point forecast of p is given in (2-2)
t t
with the associated forecast uncertainty θσ2. Following Wallis (2004), we can “mix”
z
the individual distributions to obtain an aggregate forecast distribution and its variance.
Or equivalently we can calculate the variance as the sum of average individual forecast
error variance and forecast disagreement.9 This can be done more easily in the current
context. Using previous notations, we obtain the average of individual forecast
uncertainties of p as
t
1 1
Var= ∑var(p |I (z))= ∑θσ2 =θσ2 (2-3)
N t t N z
z z
where N is the number of forecasters.

9 The decomposition of the variance of the aggregate distribution as the sum of average individual forecast
error variance and forecast disagreement is discussed in Lahiri et al. (1988) and Wallis (2004). Giordani
and Söderlind (2003) provided an alternative justification for the decomposition which is based on
integrating out private information to get the aggregate distribution of inflation. As pointed out by Wallis
(2004), this is an inappropriate approach because “aggregating forecasts is not the same as aggregating
information sets”.
8
Since forecasters observe different p (z), they will make different point forecasts. The
t
forecast disagreement defined as the variance of point forecasts across individuals is 10
1 1
Disag = ∑{E(p |I (z))− ∑E(p |I (z))}2
N t t N t t
z z (2-4)
1 1
⎯⎯p→ ∑E {E(p |I (z))−E [E(p |I (z))]}2 = ∑var [E(p |I (z)]
N z t t z t t N z t t
z z
By (2-2)
var [E(p | I (z))]= var [(1−θ )p +(1−θ )z+θ p ]= (1−θ )2τ2 (2-5)
z t t z z t z z t z z
1
So, the forecast disagreement can be approximated by ∑(1−θ )2τ2 and the sum of
N z z
z
average individual forecast uncertainty and forecast disagreement is approximately equal
to
1 1 ⎛ τ2σ2 σ4τ2 ⎞
Var +Disag = ∑(θσ2 +(1−θ )2τ2)= ∑⎜ z + z ⎟
N z z z z N z ⎜⎝σ2 +τz2 (σ2 +τz2)2 ⎟⎠ (2-6)
1 2τ2σ4 +τ4σ2 1
= ∑ z z =σ2 ∑[1−(1−θ )2]≤σ2
N (σ2 +τ2)2 N z
z z z
since 0≤θ ≤1.
z

Equation (2-6) shows the relationship of the sum of the average individual forecast error
variance and the forecast disagreement with the forecast uncertainty estimated from a
well-specified time series model. What we show is that typically the former is expected
to be less than the latter.11 However, we can establish an upper bound on the difference
between the two measures empirically based on the findings of previous studies.

10 Sample variance will converge in probability to the average of population variance. See Greene (2000),
page 504.
11 It is interesting to note that the so-called fan charts depicting multi-period forecast uncertainty as
generated by Bank of England and Riksbank are very similar in spirit to the variance of the aggregate
density from SPF. Starting from a base line model predictions that all share, bank economists and
forecasting experts subjectively generate these charts incorporating their individual beliefs and specialized
knowledge.
In a recent paper, Cogley et al. (2003) show that the predictive intervals estimated from a Bayesian VAR
that incorporate diverse statistical sources of uncertainty including model uncertainty, policy drift,
structural shifts, and other shocks are more diffuse than the Bank of England’s fan charts. This result is
consistent with our equation (2-6). See Garratt et al. (2003) for a similar forecasting model generating
uncertainty.
9
Cukierman and Wachtel (1979) found that σ2 /τ2, the ratio of the variance of the general
z
price level to the variance of relative prices, was always less than unity during the sample
period 1948-74 and, except for five years, it was less than 0.15. Thus, we see that a value
of σ2 /τ2 <1would imply Var+Disag to be at least 0.75σ2. In fact, Var+Disag will be
z
larger than 0.98σ2 if σ2 /τ2 <0.15. We also computed σ2 /τ2 with the annual PPI data
z z
during 1948-1989 from Ball and Mankiw (1995). These authors calculated PPI inflation
and weighted standard deviation of relative price change for each year. We calculated σ
as the standard error from an AR(1) regression of PPI inflation. We found that during this
period σ2 /τ2 is equal to 0.42 on the average, which implies that Var+Disag =0.91σ2.
z

Figure 1 about here

Figure 1 shows the extent of underestimation of the time series uncertainty by
Var+Disag as a function of σ2 /τ2. As expected, the smaller this ratio, the closer will be
z
Var+Disag to σ2. For example, if σ2 /τ2 is less than 0.1, Var + Disag >0.99σ2.
z
Intuitively, the higher the variance of individual specific shocks relative to that of the
general price level, the lower is the value of p (z) in predicting p . This is because it is
t t
harder for individuals to extract information about p from the strong background noise
t
of individual specific shocks. Equation (2-2) shows that, under this situation, individuals
will rely more heavily on p to form their forecasts. If θ =1 in (2-2), p (z) provides no
t z t
information about p , and individual forecasters will just report p as their point
t t
forecasts. Then Var+Disag will be exactly equal to σ2.

Under the assumption that the variances of individual specific shocks are the same, i.e.,
τ2 =τ2 = =τ2 =τ2, we can establish the lower bound of Var+Disag in another way.
1 2 L N
First note that if the variances of individual specific shocks are identical, (2-3), (2-4) and
(2-6) can be simplified as Var =θσ2, Disag =(1−θ)2τ2, Var+Disag =(1−(1−θ)2)σ2,
where θ=τ2 /(τ2 +σ2). After litter algebra, we can get
10