Rejoinder: Harold Jeffreys's Theory of Probability Revisited

0.08 MB·English

by  Christian P. Robert

StatisticalScience
2009,Vol.24,No.2,191–194
DOI:10.1214/09-STS284REJ
MainarticleDOI:10.1214/09-STS284
(cid:13)c InstituteofMathematicalStatistics,2009
Rejoinder: Harold Jeffreys’s Theory of
Probability Revisited
Christian P. Robert, Nicolas Chopin and Judith Rousseau
0
1
Abstract. We are grateful to all discussants of our re-visitation for
0
2 their strong support in our enterprise and for their overall agreement
with our perspective. Further discussions with them and other leading
n
a statisticians showed that the legacy of Theory of Probability is alive
J
and lasting.

8
1

1. ON BERNARDO’S COMMENTS sure as an invariant loss function, even though the
]
E necessity ofscaling this measureandof selecting the
We cannot but agree with most issues raised by
M
subsequentboundbetween acceptance andrejection
Professor Bernardo, first and foremost the impor-
. remainstrongimpedimentsagainstadoptingthisal-
t tantdistinctionbetweentestingandestimation.The
a ternative. (The fact that it depends on the sample
t multidimensional Jeffreys prior (for estimation) is
s size n is certainly a major drawback, although one
[ certainlynotformallydefinedwithinTheoryofProb-
could reasonably object that the boundbetween ac-
ability and the multiplication of cases in the book
2 ceptance and rejection associated with the Bayes
v does not help. We alas have no clear explanation as
factor should also depend on the sample size.)
8 to why most Jeffreys priors produce proper posteri-
0
ors for all datasets. While Lindley–Jeffreys’s para-
0 2. ON GELMAN’S COMMENTS
dox may beupsetting(although it mostly highlights
1
. the discrepancy between the frequentist and the
9 We first apologize to the authors of Gelman et al.
Bayesiananswers)we,however,considertheattempt
0 (2001) for not ranking them into the “classics” of
9 to create a testing Jeffreys prior in Section 5.2 as an
our first footnote. This choice was, however, delib-
0 interesting if incomplete attempt, concretized much
: erate: we wanted to stop short of comparing the
v later by Bayarri and Garcia-Donato (2007). We un-
most recent textbooks of the late 1990s (excluding
i
X derstandProfessorBernardo’spointofviewonBayes as well Robert, 1994). At a more foundational level,
factors,butstillresistthetemptation tothrowaway
r the debate about the choice of a noninformative or
a this useful tool, as discussed below in conjunction
of a weakly informative prior is endless, hopeless
withProfessorLindley’scomments.Wearenonethe-
and possibly fruitless, in that (a) there is no way a
less sympathetic to the intrinsic discrepancy mea-
single perfect noninformative prior can be adopted
by one and all except through a formal decision
Christian P. Robert is Professor, CEREMADE,
from the community to always use Jeffreys prior as
Universit´e Paris Dauphine, 75775 Paris cedex 16,
a default (in the same way the Black-and-Scholes
France e-mail: [email protected]. Nicolas
formula is used by financial analysts as a common
Chopin is Professor, CREST-ENSAE, INSEE, 92245
Malakoff cedex, France e-mail: [email protected]. ground, not as a representation of real series); and
Judith Rousseau is Professor, CEREMADE, Universit´e (b) noninformative and informative priors are not
Paris Dauphine, 75775 Paris cedex 16, France e-mail: two well-separated categories, they form a contin-
[email protected]. uum.Itseemsthusmorefruitfultotrytobuildmea-
suresthatassess of theimpactof agiven prior(or of
This is an electronic reprint of the original article
the variation of a parameter in a family of priors).
published by the Institute of Mathematical Statistics in
Statistical Science, 2009, Vol. 24, No. 2, 191–194. This The debate about complexity is more in line with
reprint differs from the original in pagination and our views: (a) similar to the notion of a universal
typographic detail. noninformative prior, a practical implementation of
1
2
C. P. ROBERT,N.CHOPIN AND J. ROUSSEAU
the Ockham’s razor does not exist; and (b) com- of this journal). That Bayesian testing, or any kind
plexity is quite a subjective factor. This is not to of testing, remains a source for discussion and fur-
say that we reject the Jeffreysian tenet that Bayes ther research is clearly illustrated by the number of
factors naturally downweight complex models with commentsandthevariety ofproposalsonthispoint.
limited supportfromthedata,sincewesupportthis Finally,thelackofdecisiontheoryisanissuethatwe
point,butrather that thesupportfor morecomplex also deplore,in agreement with ProfessorsBernardo
models should come from the prior or from a loss and Lindley as well, if not Professor Zellner.
function, rather than from the complexity-hungry
4. ON LINDLEY’S COMMENTS
likelihood.(Thesocialscientistattitudethatworries
about missing some factor could also be questioned Besides so kindly contributing to the discussion
as being too optimistic in its belief in models.) Fi- therein, Professor Lindley patiently and helpfully
nally, we concede that Bayesian data analysis may enlightened us on the construction and contents of
forceustomoveawayfromthe“ideal”standardsset Theory of Probability during the preparation of the
byHarold Jeffreys’sTheory of Probability, including paper. We are therefore deeply indebted to him for
the reliance on the Bayes factor. Bayesian model sharing so much with us. His comments bring a
criticism, indeed a major direction in Gelman et al. unique perspective to the discussion, both from his-
torical and foundational viewpoints. As a witness
(2001), is still in its infancy and would correctly re-
of the early developments of Theory of Probabil-
quire more emphasis in our papers and in our prac-
ity,ProfessorLindleyexposes thephilosophical cum
tice! As put by Professor Gelman, we need to learn
practical reasons for the composition of this book.
more from “the failures of a statistical model’s at-
The point about Section 3.10 and the integration
tempt to capture reality.”
overthesamplespacewasmissedinouranalysisbut
is indeed crucial in its link with the likelihood prin-
3. ON KASS’ COMMENTS
ciplethatdoesnotappearperseinTheory of Proba-
We are grateful to Professor Kass for his com- bility. Nowadays, this is certainly themost standard
ments that follow a talk given during the Harold example that illustrates how the principle for con-
Jeffreys’s Theory of Probability anniversary session structing Jeffreys’s priors may violate the likelihood
at the O-Bayes 2009 meeting. Thereis actually very principle (Berger and Wolpert, 1988). (The opposi-
little we can disagree with in these comments which tion withdeFinetti’s perspectiveisalso worthnotic-
show a deep and scholarly knowledge of Theory of ing, since they approached Bayesian statistics from
fundamentally different perspectives, even though
Probability andexposeourneedtopursueourstudy
their respective books share the same title.)
of this profound book.
The fact that uncertainty must be analyzed in
The connection with geometry was bound to be
probabilistic terms is certainly a driving force in
part of Professor Kass’ comments and we do agree
Theory of Probability and a convincing reason to
with the essential feature of looking for orthogonal
follow Bayesian ways. Wecompletely agree thatthis
parameterisation, a point which, in our awkward
formalizationisoneofHaroldJeffreys’sgreatinputs.
phrasing, we would relate to the search for a con-
Once again, the other fundamental input stressed
stant information parameterization. We also appre-
both by Professor Lindley and ourselves is the com-
ciatetheemphasisonLaplace’sapproximationsthat
pleteformalization ofacoherentapproachtotesting
permeate the book and provide an early link with
via Bayes factors. Professor Lindley is 100% correct
Bayesian asymptotics. The epistemological implica-
in his assessment of the opposition of this view with
tions of Theory of Probability are certainly worth
Popper’s and of its persistence (Templeton, 2008):
stressing (a point also made by Professor Zellner)
rejecting a model based on its “falsity” is only feasi-
if only because Harold Jeffreys was first and fore-
blewhenconsideringtheavailablealternatives.That
most a physicist who developed his own statistical Theory of Probability does not directly dwell on de-
tools to deal with his own physics problems. The cisions is clearly a feature of the time, even though
specificpointsmadebyProfessorKassaboutthena- Keynes had opened the way a few years earlier, but
ture of statistical models would be worth emphasis- thisdidnotpreventaformalizationofBayesiantest-
ing during any course in applied and even method- ing procedures that proved itself compatible with
ological statistics (as are the central discussions by “0–1” loss functions, thus showing the insight in
Erich Lehmann and David Cox in the 1990 volume Harold Jeffreys’s intuitions.
3
REJOINDER
5. ON SENN’S COMMENTS Theory of Probability with the philosophy of science
at the time it was written. This is not to say we
Given the tone of some earlier comments of Pro-
missedtheglobalimpactofTheory of Probability on
fessor Senn on the Bayesian paradigm, we must ac-
scientific modeling and its definition of induction, a
knowledge our pleasant surprise at his conciliatory
point already stressed by Professor Kass, because it
toneinthesecomments.Thankfully,thebarbedpar-
obviously represents the major impact of the book,
odyofHarold Jeffreys’smostquotedsentencesome-
but the style of the discussions about the axiomatic
how re-establishes the balance! We are quite grate-
fultoProfessorSennforhislaudatoryremarks,even nature of probability and our lack of background in
though we must acknowledge that our copies of this area led us to bypass them to focus on the link
Harold Jeffreys’s Theory of Probability are also full with modern Bayesian statistics. (Neither does the
of pencil annotations and question marks, and also “proof” of Bayes’ theorem strike us as ultimately
that it took two series of lectures to achieve this necessary, once the axiomatic definition of probabil-
incomplete state of awareness. We furthermore en- ity is agreed upon.)The coherence of the system for
joyed the mention that Harold Jeffreys considered scientific induction presented in Theory of Probabil-
Bayesian significance tests as the most important ity is what struck us the most in Theory of Proba-
part of Theory of Probability, since this agrees with bility, even though we presumably skimmed too fast
both Professor Lindley’s and our perceptions. over this point.
We dearly appreciate the further historical de- Asalreadynoted(withadifferenttwist)inthedis-
tailsprovidedbyProfessorSenn’scomments,partic- cussion about Professor Gelman’s comments, there
ularly in that the exchange between Ronald Fisher is no end to the debate about non-informative pri-
and Harold Jeffreys is represented in much less a ors and, while Professor Zellner’s maximal data in-
controversial tone that we could have believed! [The
formation prior is an interesting alternative to Jef-
first author also commented on Berger, Bernardo
freys’s, Laplace’s and Haldane’s solutions, there is
and Sun (2009) about the particular matter of the
no reason to believe the community as a whole will
Law of Succession and so we do not need to re-
eventually agree upon this point. We obviously ap-
peat the comments here.] Similarly, the confusion
preciate the derivation of this prior based on a spe-
about Bernoulli shows how amateurish is our at-
cific information criterion developed by Professor
tempt at Science History. We are equally grateful to
Zellner. In a historical perspective, it may well be
Professor Senn for pointing out Bartlett’s connec-
that the notions of “objective” or “noninformative”
tion, as we must confess we were not even aware of
are not appropriate for the (Statistics of the) mid-
it! When reading Bartlett’s comments, we came to
1930s.
realize his contribution to the exclusion of improper
The conclusion presented by Professor Zellner re-
priors for Bayes factors, as analyzed in deeper de-
produces Seymour Geisser’s assessment of Theory
tails by Bickel and Ghosh (1990).
of Probability, for which we are both grateful and in
6. ON ZELLNER’S COMMENTS complete agreement.
Unsurprisingly,ProfessorZellner’scomments—that
7. CONCLUSION
hedelivered quite enthusiastically duringhis lecture
at O-Bayes 2009—are opening new vistas on The- We are most grateful to the contributors for their
ory of Probability, while differing from our analysis lively discussions, which illustrate how influential
on several points. The first issue is that Theory of Jeffreys’sideasstillaretoday.Maybethemoststrik-
Probability was aimed at scientists at large, while ing aspect in Theory of Probability is Harold Jef-
wereaditasstatisticians. Thisisunavoidable,given freys’s intuition that a completely coherent system
our background and, further, we doubt many non- could be designed for Bayesian analysis, a system
statisticians would have the time and the will to go upon which we are still building today.
through Theory of Probability. Unfortunately, most
of them seem to eschew modern Bayesian introduc-
ACKNOWLEDGMENTS
tions to the benefit of shorter reviews published in
their own discipline. We completely agree with Pro- This reply was written after the first author at-
fessor Zellner that we failed to understand the his- tended both the O-Bayes 2009 and MaxEnt 2009
torical undercurrents explaining the connection of conferences. He is grateful to the speakers in the
4
C. P. ROBERT,N.CHOPIN AND J. ROUSSEAU
special Harold Jeffreys’s Theory of Probability ses- R. Acad. Cienc. Exactas F´ıs. Nat. Ser. A Mat. 103 125–
sion at O-Bayes 2009 and to the participants from 135. MR2535278
Bickel, P. and Ghosh, J. (1990). A decomposition for
those conferences who offered comments on the pa-
the likelihood ratio statistic and the Bartlett correction—
per or simply support for the project. a Bayesian argument. Ann. Statist. 18 1070–1090.
MR1062699
Gelman, A., Carlin, J. Stern, H. and Rubin, D. (2001).
REFERENCES Bayesian Data Analysis, 2nd ed.Chapman and Hall, New
York.
Bayarri, M. and Garcia-Donato, G. (2007). Extending Robert, C. (1994). The Bayesian Choice. Springer, New
conventionalpriorsfortestinggeneralhypothesesinlinear York.MR1313727
models. Biometrika 94 135–152. MR2367828 Templeton, A. (2008). Statistical hypothesis testing in in-
Berger, J. and Wolpert, R. (1988). The Likelihood Prin- traspecific phylogeography: Nested clade phylogeographi-
ciple, 2nd ed. IMS,Hayward, CA. calanalysisvs.approximateBayesiancomputation.Molec-
Berger, J., Bernardo, J. andSun, D.(2009). Naturalin- ular Ecology 18 319–331.
duction: An objective Bayesian approach. RACSAM Rev.