Statistical Challenges for Searches for New Physics at the LHC

0.26 MB·English

by  Kyle Cranmer

STATISTICAL CHALLENGES FOR SEARCHES FOR NEW PHYSICS AT THE LHC
6
0
0 KYLE CRANMER
2
Brookhaven National Laboratory, Upton, NY 11973, USA

n e-mail: [email protected]
a
J
BecausetheemphasisoftheLHCison5σdiscoveriesandtheLHCenvironmentinduceshighsystematicerrors,many
4
of the common statistical procedures used in High Energy Physics are not adequate. I review the basic ingredients

ofLHCsearches,thesourcesofsystematics,andtheperformanceofseveralmethods. Finally,Iindicatethemethods
]
n thatseemmostpromisingfortheLHCandareasthatareinneed offurtherstudy.
a
-
a 1 Introduction Theonlyparticleofthestandardmodelthathas
t notbeenobservedistheHiggsboson,whichiskeyfor
a
d TheLargeHadronCollider(LHC)atCERNandthe the standard model’s description of the electroweak
s. two multipurpose detectors, Atlas and CMS, have interactions. The mass of the Higgs boson, mH, is
c been built in order to discover the Higgs boson, if it a free parameter in the standard model, but there
i
s exists, and explore the theoretical landscape beyond exist direct experimental lowerbounds and more in-
y
h theStandardModel.1,2 TheLHCwillcollideprotons directupperbounds. OncemH isfixed,thestandard
p with unprecedented center-of-mass energy (√s=14 model is a completely predictive theory. There are
[ TeV)andluminosity(1034cm−2s−1);theAtlasand numerousparticle-levelMonteCarlogeneratorsthat
2 CMS detectors will record these interactions with can be interfaced with simulations of the detectors
v 108 individual electronic readouts per event. Be- topredicttherateanddistributionofallexperimen-
8 ∼
cause the emphasis of the physics programis ondis- tal observables. Because of this predictive power,
2
0 coveryand the experimental environmentis so com- searchesfortheHiggsbosonarehighlytunedandof-
1 plex,theLHCposesnewchallengestoourstatistical ten employ multivariate discriminationmethods like
1
methods – challenges we must meet with the same neuralnetworks,boosteddecisiontrees,supportvec-
5
0 vigor that led to the theoretical and experimental tor machines, and genetic programming.3,4,5
/ advancements of the last decade. While the Higgs boson is key for understand-
s
c In the remainder of this Section, I introduce the ing the electroweakinteractions, it introduces a new
i
s physics goals of the LHC and most pertinent factors problem: i.e. the hierarchy problem. There are
y
that complicate data analysis. I also review the for- several proposed solutions to the problem, one of
h
p mal link and the practical differences between confi- which is to introduce a new fundamental symmetry,
: dence intervals and hypothesis testing. called supersymmetry (SUSY), between bosons and
v
i InSec.2,theprimaryingredientstonewparticle fermions. In practice, the minimal supersymmetric
X
searches are discussed. Practical and toy examples extension to the standard model (MSSM), with its
r
a are presented in Sec. 3, which will be used to assess 105 parameters, is not so much a theory as a theo-
themostcommonmethodsinSec.4. Theremainder retical framework.
of this paper is devoted to discussion on the most They key difference between SUSY and Higgs
promising methods for the LHC. searches is that, in most cases, discovering SUSY
will not be the difficult part. Searches for SUSY
often rely on robust signatures that will show a de-
1.1 Physics Goals of the LHC
viation from the standard model for most regions of
the SUSY parameter space. It will be much more
Currently, our best experimentally justified model
challenging to demonstrate that the deviation from
for fundamental particles and their interactions is
thestandardmodelisSUSYandtomeasurethefun-
the standard model. In short, the physics goals of
damental parameters of the theory.6 In order to re-
the LHC come in two types: those that improve our
strict the scope of these proceedings, I shall focus
understandingofthestandardmodel,andthosethat
LHC Higgs searches, where the issues of hypothesis
go beyond it.
testing are more relevant.
1
2
1.2 The Challenges of LHC Environment In order to augment the simulated data chain,
most searches introduce auxiliary measurements to
The challenges of the LHC environment are mani- estimate their backgrounds from the data itself. In
fold. The first and most obvious challenge is due some cases, the background estimation is a simple
to the enormous rate of uninteresting background sideband,butinothersthelinkbetweentheauxiliary
events from QCD processes. The total interaction measurement to the quantity of interest is based on
ratefor the LHC is oforder109 interactionsper sec- simulation. This hybrid approach is of particular
ond;therateofHiggsproductionisabouttenorders importance at the LHC.
ofmagnitudesmaller. Thus,tounderstandtheback- Whilemanyoftheissuesdiscussedabovearenot
ground of a Higgs search, one must understand the unique to the LHC, they are often more severe. At
extreme tails of the QCD processes. LEP, it was possible to generate Monte Carlo sam-
Compoundingthedifficultiesduetotheextreme plesoflargersizethanthecollecteddata,QCDback-
rate is the complexity of the detectors. The full- groundsweremoretame,andmostsearcheswerenot
fledgedsimulationofthedetectorsisextremelycom- systematics-limited. TheTevatronhasmuchmorein
putationally intensive, with samples of 107 events common with the LHC; however, at this point dis-
taking about a month to produce with computing covery is less likely, and most of the emphasis is on
resourcesdistributedaroundtheglobe. Thiscompu- measurements and limit setting.
tational limitation constrains the problems that can
been addressed with Monte Carlo techniques.
1.3 Confidence Intervals & Hypothesis Testing
Theoretical uncertainties also contribute to the
challenge. The background to many searches re- The last several conferences in the vein of PhyStat
quires calculations at, or just beyond, the state- 2005 have concentrated heavily on confidence inter-
of-the-art in particle physics. The most common vals. Inparticular,95%confidenceintervalsforsome
situation requires a final state with several well- physics parameter in an experiment that typically
separated high transverse momentum objects (e.g. hasfewevents. Morerecently,therehasbeenalarge
tt¯jj blν¯bjjjj), in which the regions of phys- effortinunderstandinghowtoincludesystematicer-
→
ical interest are not reliably described by leading- rorsandnuisanceparametersintothesecalculations.
order perturbative calculations (due to infra-red LHC searches, in contrast, are primarily inter-
and collinear divergences), are too complex for the ested in 5σ discovery. The 5σ discovery criterion is
requisite next-to-next-to-leading order calculations, somewhat vague, but usually interpreted in a fre-
and are not properly described by the parton- quentist sense as a hypothesis test with a rate of
shower models alone. Enormous effort has gone Type I error α=2.85 10−7.
·
intoimprovingthesituationwithnext-to-leadingor- There is a formal link between confidence inter-
der calculations and matrix-element–parton-shower vals and hypothesis testing: frequentist confidence
matching.7,8 While these new tools are a vast im- intervalsfromtheNeymanconstructionareformally
provement, the residual uncertainties are still often inverted hypothesis tests. It is this equivalence that
dominant. links the Neyman-Pearson lemmaa to the ordering
Uncertainties from non-perturbative effects are rule used in the unified method of Feldman and
also important. For some processes, the relevant Cousins.10Furthermore,thisequivalencewillbevery
regions of the parton distribution functions are not usefulintranslatingourunderstandingofconfidence
well-measured (and probably will not be in the first intervals to the searches at the LHC.
few yearsof LHC running), whichleadto uncertain- In some cases, this formal link can be mislead-
tiesinrateaswellastheshapeofdistributions. Fur- ing. In particular, there is not always a continuous
thermore,thevariousunderlying-eventandmultiple- parameter that links the fully specified null hypoth-
interaction models used to describe data from pre- esis H0 to the fully specified alternate H1 in any
vious colliders show large deviations when extrap-
olated to the LHC.9 This soft physics has a large aThe lemma states that, for a simple hypothesis test of size
αbetween anullH0 andanalternate H1,the mostpowerful
impact on the performance of observables such as
criticalregionintheobservablexisgivenbyacontourofthe
missing transverse energy. likelihoodratioL(x|H0)/L(x|H1).
3
Ingeneralthe theoreticalmodel fora new parti-
e
gnificanc A(∫ nLTo dLKtA -=fSa 3c0to frbs-)1 HtHHtH → →→(H g g ZW →ZW ( *b)( *b ) )→ → 4 lln ln cdlaerhdamssoodmeleHfriegegps,aroanmlyettehrse.mInastshemcHaseisofutnhkensotwann-.
nal si 102 qqqqHH →→ qqqq Wtt W(*) ForSUSYscenarios,theHiggsmodelisparametrized
Sig Total significance bytwoparameters: mAandtanβ. Typically,theun-
knownvariablesarescannedandahypothesistestis
performed for each value of these parameters. The
resultsfromeachofthe searchchannelscanbe com-
10 bined to enhance the power of the search, but one
must take care of correlations among channels and
ensure consistency.
Thefactthatonescansoverthe parametersand
performsmanyhypothesistests increasesthe chance
1 100 120 140 160 180 200 thatone finds atleastonelargefluctuationfromthe
mH (GeV/c2) null-hypothesis. Some approaches incorporate the
number of trials explicitly,11 some approaches only
Figure 1. Expected significance as a function of Higgs mass
fortheAtlasdetector with30fb−1 ofdata. focusonthemostinterestingfluctuation,12andsome
see this heightened rate of Type I error as the moti-
vation for the stringent 5σ requirement.13
physically interesting or justified way. Furthermore,
the performance of a method for a 95% confidence
interval and a 5σ discovery can be quite different. 2.2 Discriminating Variables & Test Statistics
Typically,newparticlesareknowntodecaywithcer-
2 The Ingredients of an LHC Search
taincharacteristicsthatdistinguishthesignalevents
fromthoseproducedbybackgroundprocesses. Much
In order to assess the statistical methods that are
of the work of a search is to identify those observ-
availableanddevelopnewonessuitedfortheLHC,it
ables and to construct new discriminating variables
is necessaryto be familiarwith the basic ingredients
(generically denoted as m). Examples include an-
of the search. In this section, the basic ingredients,
gles between particles, invariant masses, and parti-
terminology, and nomenclature are established.
cle identification criterion. Discriminating variables
areusedin twodifferentways: todefine a signal-like
2.1 Multiple Channels & Processes
region and to weight events.
Almost all new particle searches do not observe the The usage of discriminating variables is related
particle directly, but through the signatures left by to the test statistic: the real-valued quantity used
the decay products of the particle. For instance, the to summarize the experiment. The test statistic is
Higgs boson will decay long before it interacts with thoughtofas being orderedsuchthateither largeor
thedetector,butitsdecayproductswillbedetected. small values indicate growing disagreementwith the
In many cases, the particle canbe producedand de- null hypothesis.
cayinmanydifferentconfigurations,eachofwhichis A simple “cut analysis” consists of defining a
called a search channel (see Tab. 1). There are may signal-like region bounded by upper- and lower-
be multiple signal and background processes which valuesofthesediscriminatingvariablesandcounting
contributetoeachchannel. Forexample,inH γγ, events in that region. In that case, the test statistic
→
the signal could come from any Higgs production issimplythenumberofeventsobservedinthesignal
mechanism and the background from either contin- like region. One expects b background events and s
uum γγ production or QCD backgroundswhere jets signal events, so the experimental sensitivity is op-
fakephotons. Eachoftheseprocesseshavetheirown timized by adjusting the cut values. More sophisti-
rates, distributions for observables, and uncertain- catedtechniquesusemultivariatealgorithms,suchas
ties. Furthermore, the uncertainties between pro- neural networks, to define more complicated signal-
cesses may be correlated. likeregions,buttheteststatisticremainsunchanged.
4
In these number counting analyses,the likelihood of
observing n events is simply given by the Poisson
model. Density
ttericbhunTtihiqoeunree.oafrIenthexeptaerdntisiscicournliasmr,itnoiafttihnoingsenvkuanmroibawbesrle-ctohmuentdifniosgr- Probability
the background-only (null) hypothesis, f (m), and
b
the signal-plus-background (alternate) hypothesis,
f (m) = [sf (m)+bf (m)]/(s+b), then there is Neural Network Output
s+b s b
a more powerful test statistic than simply counting Figure2. Thedistributionofaneuralnetworkoutputforsig-
events. This is intuitive, a well measured ’golden nalevents. Thehistogramisshowntogether withfˆ1(m).
event’ is often more convincing than a few messy
ones. Following the Neyman-Pearson lemma, the
where
most powerful test statistic is
1/5
4 σ
L(mH ) h(m )= n−1/5, (4)
Q = | 1 (1) i 3 fˆ(m )
L(mH0) (cid:18) (cid:19) r 0 i
|
= Ni chanPois(ni|si+bi) nji sifs(misji)++bbiifb(mij) σneilsftuhnecsttiaonnd(aursdudaellvyiatthioennoofr{mxai}l,dKist(rxi)buistsioonm)e, kaenrd-
Q Ni chanPois(ni|bQi) njifb(mij) fˆ0(x) is the fixed kernel estimate given by the same
(ni denotes eQvents in ith channelQ) or equivalently equation but with a fixed h(mi)
q =lnQ= s +Nchan ni ln 1+ sifs(mij) . h∗ = 4 1/5σn−1/5. (5)
− tot b f (m ) 3
i j (cid:18) i b ij (cid:19) (cid:18) (cid:19)
X X
(2) The solid line in Fig. 2 shows that the method
The test statistic in Eq. 2 was used by the LEP (withmodified-boundarykernels)worksverywellfor
Higgs Working Group (LHWG) in their final results shapes with complicated structure at many scales.
on the search for the Standard Model Higgs.14
Atthispoint,therearetwolooseends: howdoes 2.4 Numerical Evaluation of Significance
one determine the distribution of the discriminating
Given, f (m) and f (m) the distribution of q(x) can
s b
variables f(m), and how does one go from Eq. 2 to
be constructed. For the background-only hypothe-
the distribution of q for H and H . These are the
0 1 sis, f (m) provides the probability of corresponding
b
topics of the next subsections.
values of q needed to define the single-eventpdf ρ .b
1
2.3 Parametric and Non-Parametric Methods ρ (q )= f (m)δ(q(m) q )dm (6)
1,b 0 b 0
−
Z
In some cases, the distribution of a discriminat-
For multiple events, the distribution of the log-
ing variable f(m) can be parametrized and this
likelihoodratiomustbeobtainedfromrepeatedcon-
parametrizationcanbejustifiedeitherbyphysicsar-
volutions of the single event distribution. This con-
guments or by goodness-of-fit. However, there are
volution can either be performed implicitly with ap-
many cases in which f(m) has a complicated shape
proximate Monte Carlo techniques,16 or analytically
not easily parametrized. For instance, Fig. 2 shows
with a Fourier transformtechnique.17 In the Fourier
thedistributionofaneuralnetworkoutputforsignal
domain, denoted with a bar, the distribution of the
events. Inthatcasekernelestimationtechniquescan
log-likelihood for n events is
be used to estimate f(m) in a non-parametric way
from a sample of events mi .15 The technique that ρn =ρ1n (7)
{ }
was used by the LHWG14 was based on an adaptive
Thus the expected log-likelihood distribution for
kernel estimation given by:
backgroundwith Poissonfluctuations inthe number
n
1 m m
fˆ(m)= K − i , (3) bTheintegralisnecessarybecausethemapq(m):m→qmay
1
nh(m ) h(m )
i i (cid:18) i (cid:19) bemany-to-one.
X
5
of events takes the form Intheabsenceofanobservationwecancalculatethe
∞ e−bbn expected CLb given the signal-plus-background hy-
ρ (q)= ρ (q) (8) pothesisistrue. Todothiswefirstmustfindtheme-
b n,b
n!
nX=0 dianofthesignal-plus-backgrounddistributionqs+b.
From these we can calculate the expected CL by
which in the Fourier domain is simply b
∗
using Eq. 11 evaluated at q =q .
s+b
ρ (q)=eb[ρ1,b(q)−1]. (9) Finally,wecanconverttheexpectedbackground
b
confidence level into an expected Gaussian signifi-
Forthesignal-plus-backgroundhypothesisweexpect cance, Zσ, by finding the value of Z which satisfies
s eventsfromthe ρ distributionandbeventsfrom
1,s 1 erf(Z/√2)
the ρ1,b distribution, which leads to the expression CLb(qs+b)= − 2 . (12)
for ρ in the Fourier domainc
s+b
where erf(Z) = (2/π) Zexp( y2)dy is a function
ρs+b(q)=eb[ρ1,b(q)−1]+s[ρ1,s(q)−1]. (10) readilyavailableinmost0numeri−callibraries. ForZ >
R
1.5, the relationship can be approximated19 as
This equation generalizes, in a somewhat obvious
way, to include many processes and channels. Z √u lnu with u= 2ln(CL √2π) (13)
b
≈ − −
Numerically these computations are carried out
with the Fast Fourier Transform (FFT). The FFT 2.5 Systematic Errors, Nuisance Parameters &
is performed on a finite and discrete array, beyond Auxiliary Measurements
whichthefunctionisconsideredtobeperiodic. Thus
Sections2.3and2.4representthestateoftheartfor
the rangeofthe ρ distributions must be sufficiently
1
HEPinfrequentisthypothesistestinginthe absence
largeto hold the resulting ρ andρ distributions.
b s+b
ofuncertaintiesonratesandshapesofdistributions.
If they are not, the “spill over” beyond the maxi-
Inpractice,thetruerateofbackgroundisnotknown
mum log-likelihood ratio q will “wrap around”
max
exactly, and the shapes of distributions are sensitive
leading to unphysical ρ distributions. Because the
to experimental quantities, such as calibration coef-
range of ρ is much larger than ρ it requires a
b 1,b
ficients and particle identification efficiencies (which
verylargenumberofsamplestodescribebothdistri-
are also not known exactly). What one would call a
butions simultaneously. The implementation of this
systematicerrorinHEP,usuallycorrespondstowhat
methodrequiressomeapproximateasymptotictech-
astatisticianwouldrefertoasanuisanceparameter.
niques that describe the scaling from ρ to ρ .18
1,b b
Dealing with nuisance parameters in searches is
The nature of the FFT results in a number of
notanew problem,butperhapsithasneverbeenas
round-off errors and limit the numerical precision
to about 10−16 – which limit the method to signif- essential as it is for the LHC. In these proceedings,
Cousinsreviewsthedifferentapproachestonuisance
icance levels below about 8σ. Extrapolation tech-
parameters in HEP and the professional statistical
niques andarbitraryprecisioncalculations canover-
literature.20 Also of interest is the classification of
come these difficulties,18 but such small p-values are
systematic errors provided by Sinervo.21 In Sec. 4,
of little practical interest.
the a few techniques for incorporating nuisance pa-
From the log-likelihood distribution of the two
rameters are reviewed.
hypotheseswecancalculateanumberofusefulquan-
From an experimental point of view, the miss-
tities. Givensome experiment with an observedlog-
∗ ing ingredientis some set ofauxiliary measurements
likelihoodratio,q ,wecancalculatethebackground-
that will constrain the value of the nuisance param-
only confidence level, CL :
b
eters. The most common example would be a side-
∞
∗ ′ ′ band measurement to fix the background rate, or
CL (q )= ρ (q )dq (11)
b b
Zq∗ some control sample used to assess particle identi-
fication efficiency. Previously, I used the variable
cPerhaps it is worth noting that ρ(q) is a complex valued M to denote this auxiliary measurement22; while
function of the Fourier conjugate variable of q. Thus nu-
Linnemann,19 Cousins,20 and Rolke, Lopez, and
mericallytheexponentiationinEq.9requiresEuler’sformula
eiθ =cosθ+isinθ. Conrad23,24 usedy. Additionally,oneneeds toknow
6
0.6 and later combined to assess correlations. The fac-
Mgg
/ d ds / dMgg = N e-a Mgg torization of the likelihood and the number of nui-
sd 0.5 a=-0.048 sance parameters included impact the difficulty of
) sig a=-0.030 implementingthevariousscenariosconsideredbelow.
s1/
( 0.4
0.3 t = s / s 3 Practical and Toy Examples
SB sig
Inthis Section, afew practicalandtoy examplesare
0.2
introduced. The toy examples are meant to provide
0.1 s s s simple scenarios where results for different methods
SB sig SB
canbeeasilyobtainedinordertoexpeditetheircom-
0 parison. The practical examples are meant to ex-
100 105 110 115 120 125 130 135 140 145 150
Mgg clude methods that provide nice solutions to the toy
examples, but do not generalize to the realistic situ-
Figure3. Thesignal-likeregion andsideband for H →γγ in
ation.
whichτ iscorrelatedtobviathemodelparametera.
3.1 The Canonical Example
the likelihood function that provides the connection
betweenthenuisanceparameter(s)andtheauxiliary Consider a number-counting experiment that mea-
measurements. sures x events in the signal-like region and y events
The most common choices for the likelihood of in some sideband. For a given background rate b in
the auxiliary measurement are L(y b) = Pois(y τb) the signal-like region, say one can expect τb events
| |
and L(y b)=G(y τb,σ ), where τ is a constantthat in the sideband. Additionally, let the rate of signal
y
| |
specifiestheratioofthenumberofeventsoneexpects eventsinthesignal-likeregions–theparameterofin-
inthesidebandregiontothenumberexpectedinthe terest – be denoted µ. The corresponding likelihood
signal-like region.d function is
A constant τ is appropriate when one simply
L (x,y µ,b)=Pois(xµ+b) Pois(y τb). (14)
countsthenumberofeventsyinan“off-source”mea- P | | · |
surement. Inamoretypicalcase,oneusesthedistri- This is the same case that was considered in
bution of some other variable, call it m, to estimate Refs. 20,22,23,24 for x,y = (10) and α = 5%.
O
the number of background events inside a range of For LHC searches, we will be more interested in
m (see Fig. 3). In special cases the ratio τ is inde- x,y = (100)andα=2.85 10−7. Furthermore,the
O ·
pendentofthemodelparameters. However,inmany auxiliary measurement will rarely be a pure number
cases(e.g. f(m) e−am),theratioτ dependsonthe counting sideband measurement, but instead the re-
∝
model parameters. Moreover, sometimes the side- sultofsomefit. Soletusalsoconsiderthelikelihood
band is contaminated with signal events, thus the function
background and signal estimates can be correlated.
L (x,y µ,b)=Pois(xµ+b) G(y τb,√τb). (15)
These complications are not a problem as long as G | | · |
they are incorporated into the likelihood.
Asaconcreteexampleintheremainingsections,
The number of nuisance parameters and aux-
let us consider the case b = 100 and τ = 1. Opera-
iliary measurements can grow quite large. For in-
tionally,onewouldmeasureyandthenfindthevalue
stance, the standard practice at BaB¯ar is to form
x (y) necessary for discovery. In the language of
crit
very large likelihood functions that incorporate ev-
confidence intervals, x (y) is the value of x nec-
crit
erything from the parameters of the unitarity tri-
essary for the 100(1 α)% confidence interval in µ
angle to branching fractions and detector response. −
to exclude µ = 0. In Sec. 4 we check the coverage
0
These likelihoods are typically factorized into multi-
(TypeIerrororfalse-discoveryrate)forbothL and
P
ple pieces, which are studied independently at first
L .
G
dNote that Linnemann19 used α = 1/τ instead, but in this Linnemann reviewed thirteen methods and
paperαisreservedfortherateofTypeIerror. eleven published examples of this scenario.19 Of the
7
VV110000 ttH(120)
ee
0 G0 G 8800 ttttjbjb (QCD) M (pb/GeV) BBooxrn
/ 1/ 1 ttbb (EW) sd /d
s s 6600
tt
nn
ee
vv 4400
EE
2200
00
00 5500 110000 115500 220000 225500 330000 335500 440000
mm ((GGeeVV))
Mass (GeV)
bbbb
Figure 5. Two plausible shapes for the continuum γγ mass
Figure4. Thebbinvariant massspectrum fortt¯H signaland spectrumattheLHC.
backgroundprocessesatAtlas.
theshapeofthesignal-plus-backgrounddistribution.
publishedexamples,onlythree(theonefromhisref-
Furthermore,theoreticaluncertaintiesandb-tagging
erence18andthetwofrom19)arenearthe rangeof
uncertaintiesaffecttheshapeofthebackground-only
x,y, and α relevant for LHC searches. Linnemann’s
spectrum. In this case the incorporation of system-
review asks an equivalent question posed in this pa-
atic error on the background rate most likely pre-
per, but in a different way: what is the significance
cludes the expected significance ofthis channelfrom
(Z in Eq. 12) of a given observation x,y.
ever reaching 5σ.
Similarly, the H γγ channel has uncertainty
→
3.2 The LHC Standard Model Higgs Search in the shape of the mγγ spectrum from background
processes. One contribution to this uncertainty
The search for the standard model Higgs boson is
comes from the electromagnetic energy scale of the
by no means the only interesting search to be per-
calorimeter (an experimental nuisance parameter),
formedatthe LHC,butitis oneofthe moststudied
while another contribution comes from the theoreti-
and offers a particularly challenging set of channels
caluncertaintyinthecontinuumγγproduction. Fig-
to combine with a single method. Figure 1 shows
ure 5 shows two plausible shapes for the m spec-
γγ
theexpectedsignificanceversustheHiggsmass,m ,
H trum from “Born” and “Box” predictions.
for severalchannels individually and in combination
for the Atlas experiment.25 Two mass points are
4 Review of Methods
considered in more detail in Tab. 1, including re-
sults from Refs.1,25,26. Some of these channels will
Based on the practical example of the standard
most likely use a discriminating variable distribu-
model Higgs search at the LHC and the discussion
tion, f(m), to improve the sensitivity as described
in Sec. 2, the list of admissible methods is quite
in Sec. 2.3. I have indicated the channels that I sus-
short. Of the thirteen methods reviewed by Linne-
pectwilluse this technique. Roughestimates onthe
mann, only five are considered as reasonable or rec-
uncertainty in the background rate have also been
ommended. These can be divided into three classes:
tabulated, without regard to the classification pro-
hybridBayesian-frequentistmethods,methodsbased
posed by Sinervo.
ontheLikelihoodPrinciple,andfrequentistmethods
The backgrounduncertainties for the tt¯H chan-
based on the Neyman construction.
nel have been studied in some detail and separated
into various sources.26 Figure 4 shows the m mass
bb
spectrum for this channel.e Clearly, the shape of 4.1 Hybrid Bayesian-Frequentist Methods
the background-only distribution is quite similar to
The class of methods frequently used in HEP and
eItisnotclearifthisresultisinagreementwiththeequivalent commonly referredto as the Cousins-Highlandtech-
CMSresult.27 nique (or secondarily Bayes in statistical literature)
8
Table 1. Number of signal and background events for representative Higgs search channels for two values of Higgs mass, mH,
with30fb−1ofdata. Aroughuncertaintyonthebackgroundrateisdenotedasδb/b,withoutreferencetothetypeofsystematic
uncertainty. Thetablealsoindicatesifthechannelsareexpected touseaweightf(m)asinEq.2.
channel s b δb/b dominant backgrounds use f(m) m (GeV)
H
tt¯H tt¯bb 42 219 10% tt¯jj,tt¯bb Yes 120
→ ∼
H γγ 357 11820 0.1% γγ,jγ,jj No 120
→ ∼
qqH qqττ qqllE/ 17 14 10% Z ττ, tt¯ Yes 120
T
→ → ∼ →
qqH qqττ qqlhE/ 16 8 10% Z ττ, tt¯ Yes 120
T
qqH →qqWW→∗ qqllE/ 28.5 47.4 ∼10% t→t¯,WW Yes 120
T
→ → ∼
qqH qqWW∗ qqllE/ 262.5 89.1 10% tt¯,WW Yes 170
T
→ → ∼
H ZZ 4l 7.6 3.1 1% ZZ 4l No 170
→ → ∼ →
H WW llE/ 337 484 5% Z ττ, tt¯ Yes 170
T
→ → ∼ →
are based on a Bayesian average of frequentist p- Atlas used to assess its physics potential.1 The
values as found in the first equation of Ref.28. The method not recommended by Linnemann and was
Bayesianaverageisoverthenuisanceparametersand critically reviewed in Ref.29.
weighted by the posterior P(b|y). Thus the p-value x5′ (y)=y/τ +Z y/τ(1+1/τ) (20)
ofthe observation(x ,y ) evaluatedat µis givenby crit
0 0
p
∞ 4.2 Likelihood Intervals
p(x ,y µ)= dbp(x µ,b)P(by ) (16)
0 0 0 0
| | | As Cousins points out, the professional statistics
Z0
∞
literature seems less concerned with providing cor-
= dxP(xµ,y ) (17)
| 0 rect coverageby construction, in favor of likelihood-
Zx0
based and Bayesianmethods. The likelihood princi-
where
∞ ple states that given a measurement x all inference
P(y b) P(b)
0
P(xµ,y0)= dbP(xµ,b) | (18) about µ should be based on the likelihood function
| | P(y )
Z0 0 L(xµ). When nuisance parameters are included,
TheforminEq.16,anaverageoverp-values,issimi- |
things get considerably more complicated.
lartotheformwritteninCousins&Highland’sarti-
The profile likelihood function is an attempt to
cle; and it is re-written in Eq. 17 to the form that is
eliminate the nuisance parameters from the likeli-
morefamiliartothosefromLEPHiggssearches.16,17
hood function by replacing them with their condi-
Actually, the dependence on y and the Bayesian
0 tional maximum likelihood estimates (denoted, for
prior P(b) shown explicitly in Eq. 18 is often not
example,ˆˆb ). The profile likelihood for L in Eq. 14
appreciated by those that use this method. P
The specific methods that Linnemann considers is given by L(x,y µ0,ˆˆb(µ0)), with
|
correspond to different choices of Bayesian priors. ˆˆb(µ ) = x+y−(1+τ)µ0 (21)
The mostcommoninHEPisto ignorethe priorand 0 2(1+τ)
use a truncated Gaussian for the posterior P(by ),
| 0 (x+y (1+τ)µ0)2+4(1+τ)yµ0
which Linnemann calls Z . For the case in which + − .
N 2(1+τ)
the likelihood L(y b) is known to be Poisson, Linne- p
| The relevant likelihood ratio is then
man prefers to use a flat prior, which gives rise to a
Gamma-distributed posterior and Linnemann’s sec- L(x,y µ ,ˆˆb(µ ))
0 0
λ (µx,y)= | , (22)
ond preferred method ZΓ, which is identical to the P | L(x,y µˆ,ˆb)
ratio of Poisson means Z and can be written in |
Bi
terms of (in)complete beta functions as19 where µˆ andˆb are the unconditional maximum like-
lihood estimates.
Z =Z =B(1/(1+τ),x,y+1)/B(x,y+1). (19)
Γ Bi Oneofthestandardresultsfromstatisticsisthat
The method Linnemann calls Z5′ can be seen as an the distribution of 2lnλ converges to the χ2 dis-
−
approximation of Z for large signals and is what tribution with k degrees of freedom, where k is the
N
9
30 son means32 and Cousins’ improvement33 are actu-
log ally special cases of the Neyman construction with
-2 l25 nuisanceparameters(withandwithoutconditioning,
respectively).
Of course,the Neyman constructiondoes gener-
20 X=185, Y=100, t=1
alize to the more complicatedcases discussedabove.
Twoparticulartypesofconstructionshavebeenpre-
15
sented, both of which are related to the profile like-
lihood ratio discussed in Kendall’s chapter on likeli-
10
hoodratiotests &testefficiency.34 Thisrelationship
often leads to confusion with the profile likelihood
5
intervals discussed in Sec. 4.2.
The first method is a full Neyman construction
0
0 20 40 60 80 100 120 140 160 over both the parameters of interest and the nui-
m
sance parameters, using the profile likelihood ratio
asanorderingrule. Usingthis method,the nuisance
Figure6. Theprofilelikelihoodratio−2lnλversusthesignal
strengthµfory=100,τ =1,andx=xcrit(y)=185. parameter is “projected out”, leaving only an inter-
val in the parameters of interest. I presented this
method atPhyStat2003in the contextofhypothesis
number of parameters of interest. In our example testing,f and similar work was presented by Punzi
k = 1, so a 5σ confidence interval is defined by the at this conference.22,35 This method provides cover-
set of µ with 2lnλ(µx,y) < 25. Figure 6 shows agebyconstruction,independentoftheorderingrule
− |
the graph of 2lnλ(µx,y) for y = 100 at the criti- used.
− |
cal value of x for a 5σ discovery. The motivation for using the profile likelihood
At PhyStat2003, Nancy Reid presented var- ratioasateststatisticistwofold. First,itisinspired
ious adjustments and improvements to the pro- by the Neyman-Pearson lemma in the same way as
file likelihood which speed asymptotic convergence the Feldman-Cousins ordering rule. Secondly, it is
properties.30 Cousins considers these methods in independent of the nuisance parameters; providing
more detail from a physicist perspective.20 some hope of obtaining similar tests.g Both Punzi
Only recently was it generally appreciated that andmyself found a needto performsome “clipping”
themethodofMinuit31commonlyusedinHEPcor- to the acceptance regions to protect from irrelevant
respondsto the profile likelihoodintervals. The cov- values of the nuisance parameters spoiling the pro-
erage of these methods is not guaranteed, but has jection. For this technique to be broadly applica-
been studied in simple cases.23,24 These likelihood- ble, somegeneralizationofthis clipping procedureis
based techniques are quite promising for searches at needed and the scalability with the number of pa-
the LHC, but their coverage properties must be as- rameters must be addressed.h
sessed in the more complicated context of the LHC The second method, presented by Feldman at
with weighted events and several channels. In par- the Fermilab conference in 2000, involves a Ney-
ticular,the distributionofq inEq.10isoftenhighly manconstructionovertheparametersofinterest,but
non-Gaussian. the nuisance parameters are fixed to the conditional
maximum likelihood estimate: a method I will call
the profile construction. The profile construction is
4.3 The Neyman Construction with Systematics
an approximation of the full construction, that does
Linnemann’s preferred method, Z , is related to
Bi fInsimplehypothesistestingµisnotacontinuousparameter,
the familiar result on the ratio of Poisson means.32
butonlytakes onthevalues µ0=0orµ1=s.
Unfortunately, the form of Z is tightly coupled gSimilartests arethoseinwhichthecriticalregionsofsizeα
Bi
to the form of Eq. 14, and can not be directly ap- areindependent ofthenuisance parameters. Similartests do
notexistingeneral.
pliedtothemorecomplicatedcasesdescribedabove. hA Monte Carlo sampling of the nuisance parameter space
However, the standard result on the ratio of Pois- couldbeusedtocurbthecurseofdimensionality.22
10
not necessarily cover. To the extent that the use of contours for b =100, critical regions for t = 1
true
theprofilelikelihoodratioasateststatisticprovides y130
similartests,theprofileconstructionhasgoodcover-
120 No Systematics
age properties. The main motivation for the profile Z5¢
constructionisthatitscaleswellwiththenumberof 110 ZN
l profile
G
nuisance parameters and that the “clipping” is built 100 l profile
P
in(onlyone valueofthe nuisanceparametersis con- ad hoc
90 correct coverage
sidered).
It appears that the chooz experiment actually 80
performedboththefullconstruction(called“FCcor-
70
rect syst.”) and the profile construction (called “FC
60
profile”) in order to compare with the strong confi-
dence technique.36 50
Another perceived problem with the full con-
40
struction is that bad over-coverage can result from 60 80 100 120 140 160 180 200
x
the projection onto the parameters of interest. It
should be made very clear that the coverage proba- Figure7. Acomparisonofthevariousmethodscriticalbound-
bilityisafunctionofboththeparametersofinterest ary xcrit(y) (see text). The concentric ovals represent con-
and the nuisance parameters. If the data are con- toursofLG fromEq.15.
sistent with the null hypothesis for any value of the
nuisance parameters, then one should probably not
reject it. This argument is stronger for nuisance pa- The result of the full Neyman construction and
rameters directly related to the backgroundhypoth- the profile construction are not presented. The full
esis,andlessstrongforthosethataccountforinstru- Neyman construction covers by construction, and
mentation effects. In fact, there is a family of meth- it was previously demonstrated for a similar case
ods that lie between the full construction and the (b = 100, τ = 4) that the profile construction gives
profileconstruction. Perhapsweshouldpursueahy- similarresults.22Furthermore,iftheλ wereusedas
P
bridapproachinwhichtheconstructionisformedfor an ordering rule in the full construction, the critical
those parameters directly linked to the background region for b = 100 would be identical to the curve
hypothesis, the additional nuisance parameters take labeled “λ profile” (since λ actually covers).
P P
on their profile values, and the final interval is pro- It should be noted that if one knows the likeli-
jected onto the parameters of interest. hood is given by L (x,y µ,b), then one should use
G
|
thecorrespondingprofilelikelihoodratio,λ (x,y µ),
G
|
5 Results with the Canonical Example for the hypothesis test. However, knowledge of the
correct likelihood is not always available (Sinervo’s
Consider the case b = 100, τ = 1 (i.e. 10% sys- Class II systematic), so it is informative to check
true
tematic uncertainty). For each of the methods we the coverage of tests based on both λ (x,y µ) and
G
|
find the critical boundary, x (y), which is neces- λ (x,y µ) by generating Monte Carlo according to
crit P
|
sary to reject the null hypothesis µ =0 at 5σ when L (x,y µ,b)andL (x,y µ,b). Inasimilarway,this
0 G P
| |
yismeasuredintheauxiliarymeasurement. Figure7 decoupling of true likelihood and the assumed likeli-
showsthecontoursofL ,fromEq.15,andthecriti- hood (used to find the critical region) can break the
G
calboundaryfor severalmethods. The far left curve “guaranteed” coverageof the Neyman construction.
showsthe simple s/√b curveneglecting systematics. ItisquitesignificantthattheZ methodunder-
N
The far right curve shows a critical region with the covers,sinceitissocommonlyusedinHEP.Thede-
correct coverage. With the exception of the profile gree to which the method under-covers depends on
likelihood, λ , all of the other methods lie between thetruncationoftheGaussianposteriorP(by). Lin-
P
|
these two curves (ie. all of them under-cover). The nemann’stablealsoshowssignificantunder-coverage
rateofTypeI errorforthesemethods wasevaluated (overestimate ofthe significanceZ). Inorderto ob-
for L and L and presented in Table 2. tain a critical region with the correct coverage, the
G P