Predicting Box Office with and without Markets: Do Internet Users Know Anything?

40 Pages·2010·0.28 MB·English

by  Jordi Mckenzie

Predicting Box Office with and without Markets:
Do Internet Users Know Anything?

Jordi McKenzie*

May 26, 2010

Abstract

This study investigates internet users’ predictions of US box office revenues using a
market and non-market based mechanism. Using data from the popular Hollywood
Stock Exchange and the lesser known Derby game on boxofficemojo.com it is shown
that in both settings users formulate remarkably accurate predictions. Participants
appear attune to the heavy tailed nature of the industry but with a slight bias towards
over-prediction of low earning films and under-prediction of high earning films.
There is also some evidence that sequels and films featuring stars are over-predicted.
Harnessing the Derby’s detailed information of individual predictions it is also found
that individuals make more accurate predictions for big budget, wide release titles but
are less accurate on films with stars. Further, predictive accuracy improves as films
spend more time at the box office and individuals’ predictive performance improves
with experience.

Keywords: Motion pictures, fantasy game, prediction market, information
aggregation, forecasting.

JEL Classification Numbers: C93, D8, L8

* Discipline of Economics, University of Sydney, NSW, 2006. Email: [email protected].
I am extremely grateful to Elina Gilbourd for research assistance on this project.
I. INTRODUCTION
This study investigates web-based predictions of box office revenues using the well
known Hollywood Stock Exchange (HSX) and a lesser known prediction game run
through the popular movie-buff website ‘Box Office Mojo’ (boxofficemojo.com).
Accurately predicting box office returns has proven a difficult task for industry and
academics as the industry is notoriously characterised by extreme levels of
uncertainty which manifests in the famous quote of screenwriter William Goldman
who declared about the industry: “nobody knows anything” (Goldman, 1983). In
particular, the causes and consequences of the uncertainty of the box office have been
explored by De Vany and Walls (1996, 1999, 2004) who find that both revenues and
profits are well described by the stable-Paretian class of statistical distribution which
implies theoretically infinite variance in returns. Such distributions arise
predominantly because of the inherent uncertainty of audience reception and the fact
that consumer interaction causes bandwagon and leveraging effects in demand
markedly skewing distributions of returns.

Whilst many studies have pursued statistical techniques for predicting box office
(recent examples include Walls, 2005a; 2005b), others have recognised the power of
prediction markets to serve this role. In particular, a number of studies have shown
that predictions arising from the HSX are remarkably accurate even though the market
is an ‘artificial’ one in the sense that no real (i.e. monetary) incentives are provided
(see for example Pennock et al, 2001; Spann and Skiera, 2003; and Foutz and Jank,
2010). The success and popularity of the HSX market has recently led its parent
company Cantor Fitzgerald to apply to the Commodity Futures Trading Commission
(CFTC) for approval to operate a fully regulated futures exchange for domestic box
1
office receipts (DBOR) with a separate company, Veriana Networks, also filing a
similar application. At the time of writing, however, the Motion Picture Association
of America (MPAA) had attempted to block through Congress any form of trading for
box office returns citing concerns about possible market manipulation and insider
trading as key objections. On the other hand, the proponents of these exchanges argue
that such markets will allow for better risk management practices, will create new
investments opportunities, and that increased price transparency more generally will
benefit the economy. Although a full discussion of the potential benefits and costs of
a futures market for box office receipts is beyond the scope of this study, it is apparent
that broader issues relating to box office predictions are important. Specifically, is
there knowledge out there that can be tapped into in order to create a useful market?
And, if so, what sort of mechanism can achieve this?

A further practical motivation for studying box office revenue predictions derives
from the recent experience of Hollywood. The Global Financial Crisis (GFC) and its
after-math impacted heavily on Hollywood exemplified by many studio executives
losing their jobs as a consequence of poor investment decisions.1 As well financing
money which had previously come from hedge funds and other external sources
declined significantly post GFC as investors looked for more certainty in their
portfolios. Consequently Hollywood has been observed reverting to safer and
traditional formulas in film selection and production with large budget franchise
films, sequels, and TV show/comic book/computer game adaptations, etc again
popular. Such changes in tastes have also caused a number of studios to close niche
production divisions in recent times. It has also been observed that studios have

1 Between August and October 2009, Paramount, MGM, Universal, and Disney all fired top level
executives. See “Hollywood Studios in Midst of Their Own Horror Show” (LA Times, 6/10/09).
2
increasingly less reliance on stars as a means of reducing the uncertainty of the box
office as the big names fail to draw crowds.2 It would seem, at least from an
outsider’s perspective, that there is a belief in Hollywood that certain types of films
may be more predictable than others. This presents testable hypotheses relating to
which types of films are predicted more accurately in terms of observable film
characteristics relating to such things as budget, number of theatres, star power, genre,
rating, etc.

In this study, information on predicted revenues is derived from two sources – the
HSX and the Derby game run through the Box Office Mojo website. Consistent with
other studies, evidence is found that the HSX provides remarkably accurate
predictions with respect to opening weekend box office, however, it is also found that
the Derby game similarly generates extremely accurate predictions – though not quite
as accurate as the HSX – even in the absence of a market/price mechanism. It is
observed that both sets of predictions reflect the inherent skew and kurtosis which
characterises returns in the industry but that there is a tendency for small earning films
to be over-predicted and for large earning films to be under-predicted. There is also
some weak evidence suggesting that films featuring stars and which are sequels are
over-predicted and that certain genres (action, comedy, drama, and musicals) have a
tendency to be under-predicted as do PG-13 films. Comparison to a simple statistical
regression model suggests that both the HSX and Derby prediction models provide
better forecasts than does a set of explanatory film covariates including budget,
opening week theatres, star, sequel, genre and rating variables.

2 LA Times (6/10/09).
3
Given the implicit richness of the Derby data used in this study, a number of further
investigations of this predictive source are also conducted. Because individuals’
predictions are observed, a formal model of predictive accuracy based on the method
of Chen and Plott (2002) is also used to study the Derby’s predictions and it is
observed that the game aggregates information in a manner statistically consistent
with the underlying probability distribution of film success. It is also observed that
individuals formulate better predictions the more experience they acquire in the game.
The importance of individual film attributes are also considered to determine which, if
any, are important to the market in formulating accurate predictions. Consistent with
anecdotal evidence, it is found that participants find it easier to judge films with
higher budgets, and on higher screen counts, but that they less accurate on films with
stars. Further, evidence is found that individuals more accurately predict sequels,
remakes and reissues.

II. PREDICTION AND FORECASTING: RELEVANT LITERATURE
At least since the work of Hayek (1945) economists have realised that collectives
often possess the ability to formulate better predictions than any individual alone. In
recent years, prediction markets have been enjoying renewed interest as useful means
by which to aggregate widely dispersed information.3 The so-called ‘information
aggregation mechanism’ (IAM) may take a variety of forms such as a double auction
(Chen and Plott, 2002) or pari-mutuel design (Plott et al, 2003; Axelrod et al, 2009).
Whilst most prediction markets rely on offering real financial incentives to
participants, there are many examples of ‘fantasy’ markets played online which attract

3 The conceptual importance and intellectual appeal of prediction markets within the discipline has
recently prompted a number of prominent economists – including four Nobel laureates – to call on the
US Government to lower the regulatory impediments to their use stating in their conclusion:
“Prediction markets have great potential for improving economic welfare and the decisions of private
and public institutions alike” [Arrow, et al., (2007, p.4)].
4
large numbers of dedicated and enthusiastic players yet offer no real monetary
incentives – for example fantasy sports betting, or virtual stock-markets such as the
HSX. The relationship between the HSX prediction markets and actual outcomes has
been studied by a number of authors. Pennock et al. (2001) investigate, amongst
other examples, the HSX and show a high degree of correlation between actual and
predicted outcomes (correlation of 0.945 for opening weekend box office, and 0.978
for four week opening box office). Spann and Skiera (2003) also study the HSX and
similarly conclude that the market has strong predictive power but find that the
predictions are biased upwards for small earning films and biased downwards for
large earning films. Foutz and Jank (2010) find not only strong predictive power of
closing HSX prices, and that predictions accuracy is improved when the history of the
price is included using functional shape analysis techniques. Finally, in a more
general survey of prediction markets, Wolfers and Zitzewitz (2004) also note the
strong correlation of the HSX prices and actual revenues.

Although some may argue there is little use studying such ‘artificial’ markets due to
the lack of real incentives, others have observed that predictions arising within them
are similar to markets where real financial gains are attainable (Pennock, et al, 2001;
Servan-Schreiber et al, 2004). Indeed, it seems plausible that individuals who
participate in such markets and games do so for the implicit satisfaction they derive
from simply beating other participants, and it is this incentive alone that motivates
them to play seriously. Beyond questions pertaining to artificial vs. real markets,
there has also recently been questions as to whether market based mechanisms always
do provide better forecasts than other mechanisms – for example statistical
forecasting, polling or surveying. In the context of elections, Erikson and Wlezien
5
(2008) and Rothschild (2009) show the superiority of polls under certain conditions,
and in the context of football, baseball and movies Goel et al (2009) point out that the
benefits of prediction markets are often only marginal and these benefits need to be
weighed against costs before considering their implementation.

As the data used in this study represent both an artificial market (the HSX) and a non-
market based prediction technique (the Derby game) it is important to realise that
their may be inherent weaknesses in each. Yet, as the literature has documented, the
jury is still out on how important these actually are in certain situations.

III. THE HSX AND DERBY GAME
The primary prediction data used in this study come from the HSX and Box Office
Mojo’s Derby game. The HSX is a popular artificial market in which traders can
buy/sell/short/cover ‘movie stocks’ and ‘star bonds’ in a double auction environment
using imaginary ‘Hollywood Dollars’ which players are given when they create an
account. The HSX has been running since 1996 and has many thousand of registered
players. For the purposes of this study, we focus on ‘movie stocks’ which relate to a
film’s four week box office earnings. Films are typically listed on the HSX many
months (and even years) ahead of their theatrical release and as the release date
approaches traders buy and sell movie stocks at a price which represents the
prediction of the film’s four week North American box office revenue divided by
1,000,000 – for example a film trading at H$30 equates to a film earning US$30m in
its first four weeks of theatrical release.

6
The HSX market only includes films in wide release (>650 theatres), and in the case
of platform releases, the market doesn’t commence trading until the film reaches the
wide release number of theatres. On the release day of a particular film, trading is
halted until the following Monday (or Tuesday in the case of a long weekend) where a
readjustment occurs relative to the weekend’s studio reported box office estimate.
The readjustment for most films is 2.8*(three day weekend box office estimate). In
the event of a four day weekend, or in the case when a film opened on a Thursday, a
multiplier of 2.4 is used; and in the case of a five day weekend, or if a film opened on
a Wednesday, a multiplier of 2.2 is used. The fact that this adjustment takes place is
designed to realign the market price with a simple extrapolated prediction which
essentially means that the market’s price prior to the release date is in fact a prediction
of the opening weekend box office takings. For example, suppose a regular Friday
opening film had a HSX price of H$30 (i.e. the four week expectation was for the film
to make US$30m) and the opening weekend box office estimate came in at US$10m,
this would imply the HSX price is adjusted to H$28 on the following Monday.
Therefore traders who held long positions at the halt price (i.e. thought it was going to
earn more than US$30m) would be out of the money, and traders who held short
positions at the halt price (i.e. thought it was going to earn less than US$30m) would
be in the money.

The Derby game is an on-line prediction game run through the Box Office Mojo
website in which players attempt to predict weekend box office revenues.4 Unlike the
HSX, however, there is no market per se and predictions are simply the average of
individuals’ predictions. It began in March 2002 and is entered by approximately 350-

4 See http://www.boxofficemojo.com/derbygame/
7
800 players each week. Any individual can participate by registering with the website
and submitting predictions. Predictions are limited to the ten movies nominated by the
organisers each week and are generally films with the highest expected revenues.
Entries have to be submitted by midnight each Thursday for the coming weekend’s
box office (Friday-Sunday) and a valid entry must contain a prediction for each of the
ten movies. Participants are directed to a screen where they can see the average
predictions for each title and the number of valid entries received up until that time.

Participants predict the box office revenue (rounded to the nearest US$0.1m) and in
doing so implicitly pick the rank ordering. The accuracy of the prediction is judged
relative to the actual gross revenue for each title. Thus the accuracy of player i on
film j for weekend t is given as
Accuracy =1− Predicted − Actual / Actual (1)
ijt ijt jt jt
where Predicted and Actual are respectively predicted and actual gross revenues. It
should be apparent that this is one minus the absolute percentage error. Accuracy is
restricted such that Accuracy ∈[0,1], with a score of one indicating perfect
ijt
prediction. If the participant predicts more than 100% from the actual gross (i.e. above
or below) they are awarded zero. Weekly Derby winners are determined by their
overall prediction accuracy, which is the average of accuracy over the ten nominated
movies (i.e. one minus the mean absolute percentage error across ten films). Overall
champions are defined in two ways, those that dominate in terms of accumulated
points and those that dominate in terms of accuracy. To qualify for the Derby
champion by accuracy, however, players must have played in a total of at least ten
games (weeks) with one of these having occurred within the past three weeks.

8
IV DATA AND FORECAST ACCURACY
The data used in this study are derived primarily from the films played in the Derby
across 51 weeks of 2007. Given that 10 films are selected for prediction each week,
this implies that the Derby set includes 510 film/week revenue predictions. Within
this sample 176 distinct films and 2,523 individual participants are observed giving a
total of 228,686 prediction data points. Of the 176 titles, 158 of these are observed in
the first week of release which provides the set of films for comparison with the HSX
data. The sample is further reduced to 141 to only consider films opening on a
regular Friday so that the two data sets are comparable. This is done due to the fact
that films which open on a Wednesday or Thursday are treated as a four or five day
weekend for the HSX data, whereas all Derby predictions are just for three day
weekends. In this study, the HSX data reports a single number (the halt price) which
is deflated by the multiplier 2.8 to give the (three day) opening weekend prediction.
Because the Derby prediction is always a three day figure, even in the event of a
public holiday when HSX use a different multiplier for a four day weekend, the 2.8
multiplier is still used given the assumption that individuals would still scale by 2.8
after three days, even though the multiplier of 2.4 is actually used after four days by
HSX.

Across the Derby sample, the correlation between individual predictions and actual
revenues is high at 0.88 suggesting that generally participants are doing quite well.
However, when individuals’ estimates are aggregated and averaged for any particular
film/week revenue observation (i.e. 510 observations) the correlation of predicted
with actual grosses is much higher at 0.964 suggesting an aggregating effect. Indeed
the OLS regression slope coefficient of 0.994, statistically indistinguishable from one
9